Nov 02 22:45:14.944164 ip-10-0-135-250 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 22:45:14.944178 ip-10-0-135-250 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 22:45:14.944189 ip-10-0-135-250 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 22:45:14.944562 ip-10-0-135-250 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 22:45:25.192665 ip-10-0-135-250 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 22:45:25.192683 ip-10-0-135-250 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 2dab55aaa2ba4c2d851e5da4b2003631 -- Nov 02 22:47:32.375909 ip-10-0-135-250 systemd[1]: Starting Kubernetes Kubelet... Nov 02 22:47:32.723096 ip-10-0-135-250 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:47:32.723096 ip-10-0-135-250 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 22:47:32.723096 ip-10-0-135-250 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:47:32.723096 ip-10-0-135-250 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:47:32.728033 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.727935 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:47:32.728033 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728028 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:47:32.728033 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728034 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:47:32.728033 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728042 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:47:32.728294 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728048 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:47:32.728294 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728053 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:47:32.728294 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728057 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:47:32.728294 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728061 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:47:32.728294 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728065 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:47:32.728294 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728069 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:47:32.728294 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728072 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:47:32.728294 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728076 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:47:32.728294 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728080 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:47:32.728294 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728083 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:47:32.728294 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728087 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:47:32.728294 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728091 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:47:32.728294 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728095 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:47:32.728294 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728099 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:47:32.728294 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728103 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:47:32.728294 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728107 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:47:32.728294 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728120 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:47:32.728294 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728125 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:47:32.728294 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728129 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:47:32.728294 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728133 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:47:32.729100 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728136 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:47:32.729100 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728141 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:47:32.729100 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728145 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:47:32.729100 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728148 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:47:32.729100 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728153 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:47:32.729100 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728157 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:47:32.729100 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728161 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:47:32.729100 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728166 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:47:32.729100 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728171 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:47:32.729100 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728176 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:47:32.729100 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728180 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:47:32.729100 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728185 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:47:32.729100 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728189 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:47:32.729100 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728194 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:47:32.729100 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728198 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:47:32.729100 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728203 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:47:32.729100 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728207 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:47:32.729100 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728211 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:47:32.729100 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728215 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:47:32.729100 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728219 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:47:32.729644 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728224 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:47:32.729644 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728228 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:47:32.729644 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728232 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:47:32.729644 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728237 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:47:32.729644 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728241 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:47:32.729644 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728248 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:47:32.729644 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728255 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:47:32.729644 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728261 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:47:32.729644 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728266 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:47:32.729644 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728271 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:47:32.729644 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728278 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:47:32.729644 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728282 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:47:32.729644 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728287 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:47:32.729644 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728291 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:47:32.729644 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728296 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:47:32.729644 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728300 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:47:32.729644 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728303 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:47:32.729644 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728308 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:47:32.729644 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728312 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:47:32.730187 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728317 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:47:32.730187 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728322 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:47:32.730187 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728326 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:47:32.730187 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728331 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:47:32.730187 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728335 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:47:32.730187 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728339 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:47:32.730187 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728344 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:47:32.730187 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728348 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:47:32.730187 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728352 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:47:32.730187 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728356 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:47:32.730187 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728360 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:47:32.730187 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728365 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:47:32.730187 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728369 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:47:32.730187 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728373 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:47:32.730187 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728377 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:47:32.730187 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728381 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:47:32.730187 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728385 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:47:32.730187 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728390 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:47:32.730187 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728394 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:47:32.730187 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728398 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:47:32.731016 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728402 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:47:32.731016 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728407 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:47:32.731016 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728411 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:47:32.731016 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728417 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:47:32.731016 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728421 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:47:32.731016 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728425 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:47:32.731016 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728430 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:47:32.731016 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728434 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:47:32.731016 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728438 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:47:32.731016 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728442 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:47:32.731016 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728448 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:47:32.731016 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728453 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:47:32.731016 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728458 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:47:32.731016 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728462 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:47:32.731016 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728468 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:47:32.731016 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728472 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:47:32.731016 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728476 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:47:32.731016 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728500 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:47:32.731016 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728504 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:47:32.731849 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728508 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:47:32.731849 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728512 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:47:32.731849 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728517 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:47:32.731849 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.728521 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:47:32.731849 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730821 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:47:32.731849 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730836 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:47:32.731849 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730843 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:47:32.731849 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730848 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:47:32.731849 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730853 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:47:32.731849 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730857 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:47:32.731849 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730861 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:47:32.731849 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730867 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:47:32.731849 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730871 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:47:32.731849 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730876 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:47:32.731849 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730880 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:47:32.731849 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730884 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:47:32.731849 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730889 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:47:32.731849 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730894 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:47:32.731849 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730898 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:47:32.731849 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730902 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:47:32.732734 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730907 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:47:32.732734 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730911 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:47:32.732734 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730915 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:47:32.732734 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730920 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:47:32.732734 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730925 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:47:32.732734 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730929 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:47:32.732734 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730934 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:47:32.732734 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730938 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:47:32.732734 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730942 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:47:32.732734 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730946 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:47:32.732734 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730950 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:47:32.732734 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730955 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:47:32.732734 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730959 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:47:32.732734 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730963 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:47:32.732734 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730967 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:47:32.732734 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730972 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:47:32.732734 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730976 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:47:32.732734 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730980 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:47:32.732734 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730984 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:47:32.733254 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730989 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:47:32.733254 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730994 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:47:32.733254 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.730998 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:47:32.733254 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731002 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:47:32.733254 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731006 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:47:32.733254 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731012 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:47:32.733254 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731018 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:47:32.733254 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731023 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:47:32.733254 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731027 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:47:32.733254 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731031 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:47:32.733254 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731034 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:47:32.733254 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731039 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:47:32.733254 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731044 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:47:32.733254 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731048 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:47:32.733254 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731052 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:47:32.733254 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731057 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:47:32.733254 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731061 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:47:32.733254 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731067 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:47:32.733254 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731071 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:47:32.733254 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731075 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:47:32.733910 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731080 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:47:32.733910 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731084 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:47:32.733910 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731088 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:47:32.733910 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731094 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:47:32.733910 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731099 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:47:32.733910 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731104 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:47:32.733910 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731109 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:47:32.733910 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731114 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:47:32.733910 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731118 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:47:32.733910 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731122 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:47:32.733910 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731126 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:47:32.733910 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731130 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:47:32.733910 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731134 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:47:32.733910 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731138 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:47:32.733910 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731143 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:47:32.733910 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731147 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:47:32.733910 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731151 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:47:32.733910 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731155 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:47:32.733910 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731159 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:47:32.733910 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731163 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:47:32.734794 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731167 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:47:32.734794 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731172 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:47:32.734794 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731176 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:47:32.734794 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731180 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:47:32.734794 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731186 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:47:32.734794 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731190 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:47:32.734794 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731194 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:47:32.734794 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731198 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:47:32.734794 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731203 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:47:32.734794 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731208 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:47:32.734794 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731212 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:47:32.734794 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731216 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:47:32.734794 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731221 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:47:32.734794 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731225 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:47:32.734794 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731229 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:47:32.734794 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731234 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:47:32.734794 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731238 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:47:32.734794 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731242 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:47:32.734794 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731246 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:47:32.735637 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731251 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:47:32.735637 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731256 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:47:32.735637 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731260 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:47:32.735637 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731265 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:47:32.735637 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731269 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:47:32.735637 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731274 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:47:32.735637 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731282 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:47:32.735637 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731288 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:47:32.735637 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731294 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:47:32.735637 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731299 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:47:32.735637 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731307 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:47:32.735637 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.731312 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:47:32.735637 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732198 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:47:32.735637 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732209 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:47:32.735637 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732215 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:47:32.735637 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732220 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:47:32.735637 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732226 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:47:32.735637 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732231 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:47:32.736221 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732236 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:47:32.736221 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732240 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:47:32.736221 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732245 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:47:32.736221 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732250 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:47:32.736221 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732255 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:47:32.736221 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732259 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:47:32.736221 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732263 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:47:32.736221 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732268 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:47:32.736221 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732272 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:47:32.736221 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732277 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:47:32.736221 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732281 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:47:32.736221 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732285 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:47:32.736221 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732289 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:47:32.736221 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732294 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:47:32.736221 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732299 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:47:32.736221 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732303 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:47:32.736221 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732307 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:47:32.736221 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732311 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:47:32.736221 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732315 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:47:32.736221 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732319 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:47:32.736819 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732324 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:47:32.736819 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732328 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:47:32.736819 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732332 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:47:32.736819 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732336 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:47:32.736819 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732340 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:47:32.736819 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732344 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:47:32.736819 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732349 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:47:32.736819 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732353 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:47:32.736819 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732358 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:47:32.736819 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732362 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:47:32.736819 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732366 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:47:32.736819 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732371 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:47:32.736819 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732377 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:47:32.736819 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732381 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:47:32.736819 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732386 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:47:32.736819 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732391 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:47:32.736819 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732396 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:47:32.736819 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732400 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:47:32.736819 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732404 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:47:32.736819 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732408 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:47:32.737461 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732412 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:47:32.737461 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732419 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:47:32.737461 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732423 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:47:32.737461 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732427 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:47:32.737461 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732432 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:47:32.737461 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732436 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:47:32.737461 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732441 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:47:32.737461 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732445 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:47:32.737461 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732449 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:47:32.737461 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732453 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:47:32.737461 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732458 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:47:32.737461 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732462 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:47:32.737461 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732466 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:47:32.737461 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732470 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:47:32.737461 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732475 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:47:32.737461 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732499 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:47:32.737461 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732504 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:47:32.737461 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732508 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:47:32.737461 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732512 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:47:32.737461 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732516 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:47:32.738018 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732520 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:47:32.738018 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732524 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:47:32.738018 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732528 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:47:32.738018 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732532 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:47:32.738018 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732537 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:47:32.738018 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732543 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:47:32.738018 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732548 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:47:32.738018 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732552 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:47:32.738018 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732556 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:47:32.738018 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732561 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:47:32.738018 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732565 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:47:32.738018 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732569 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:47:32.738018 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732573 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:47:32.738018 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732578 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:47:32.738018 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732582 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:47:32.738018 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732587 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:47:32.738018 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732591 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:47:32.738018 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732595 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:47:32.738018 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732599 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:47:32.738527 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732603 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:47:32.738527 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732607 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:47:32.738527 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732612 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:47:32.738527 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732616 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:47:32.738527 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732620 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:47:32.738527 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732624 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:47:32.738527 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732628 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:47:32.738527 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732635 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:47:32.738527 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732642 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:47:32.738527 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732646 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:47:32.738527 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732650 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:47:32.738527 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732656 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:47:32.738527 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732661 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:47:32.738527 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732666 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:47:32.738527 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732670 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:47:32.738527 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732674 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:47:32.738527 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732678 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:47:32.738527 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732683 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:47:32.738527 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732691 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:47:32.738527 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732695 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:47:32.739052 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.732699 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:47:32.739052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733608 2578 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 22:47:32.739052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733627 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 22:47:32.739052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733637 2578 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 22:47:32.739052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733644 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 22:47:32.739052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733652 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 22:47:32.739052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733657 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 22:47:32.739052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733665 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 22:47:32.739052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733672 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 22:47:32.739052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733678 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 22:47:32.739052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733683 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 22:47:32.739052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733695 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 22:47:32.739052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733701 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 22:47:32.739052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733706 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 22:47:32.739052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733711 2578 flags.go:64] FLAG: --cgroup-root="" Nov 02 22:47:32.739052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733715 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 22:47:32.739052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733720 2578 flags.go:64] FLAG: --client-ca-file="" Nov 02 22:47:32.739052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733725 2578 flags.go:64] FLAG: --cloud-provider="external" Nov 02 22:47:32.739052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733730 2578 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 22:47:32.739052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733739 2578 flags.go:64] FLAG: --cluster-domain="" Nov 02 22:47:32.739052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733744 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 22:47:32.739052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733749 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 22:47:32.739052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733754 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 22:47:32.739052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733759 2578 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 22:47:32.739659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733766 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 22:47:32.739659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733771 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 22:47:32.739659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733776 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 22:47:32.739659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733782 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 22:47:32.739659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733787 2578 flags.go:64] FLAG: --contention-profiling="false" Nov 02 22:47:32.739659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733792 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 22:47:32.739659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733797 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 22:47:32.739659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733811 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 22:47:32.739659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733817 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 22:47:32.739659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733824 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 22:47:32.739659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733829 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 22:47:32.739659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733834 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 22:47:32.739659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733839 2578 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 22:47:32.739659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733843 2578 flags.go:64] FLAG: --enable-server="true" Nov 02 22:47:32.739659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733848 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 22:47:32.739659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733856 2578 flags.go:64] FLAG: --event-burst="100" Nov 02 22:47:32.739659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733861 2578 flags.go:64] FLAG: --event-qps="50" Nov 02 22:47:32.739659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733866 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 22:47:32.739659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733870 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 22:47:32.739659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733875 2578 flags.go:64] FLAG: --eviction-hard="" Nov 02 22:47:32.739659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733882 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 22:47:32.739659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733887 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 22:47:32.739659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733892 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 22:47:32.739659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733897 2578 flags.go:64] FLAG: --eviction-soft="" Nov 02 22:47:32.739659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733902 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 22:47:32.741029 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733906 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 22:47:32.741029 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733915 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 22:47:32.741029 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733919 2578 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 22:47:32.741029 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733923 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 22:47:32.741029 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733928 2578 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 22:47:32.741029 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733932 2578 flags.go:64] FLAG: --feature-gates="" Nov 02 22:47:32.741029 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733939 2578 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 22:47:32.741029 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733944 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 22:47:32.741029 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733950 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 22:47:32.741029 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733955 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 22:47:32.741029 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733960 2578 flags.go:64] FLAG: --healthz-port="10248" Nov 02 22:47:32.741029 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733965 2578 flags.go:64] FLAG: --help="false" Nov 02 22:47:32.741029 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733970 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:47:32.741029 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733975 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 22:47:32.741029 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733981 2578 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 22:47:32.741029 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733986 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 22:47:32.741029 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733993 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 22:47:32.741029 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.733999 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 22:47:32.741029 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734004 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 22:47:32.741029 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734008 2578 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 22:47:32.741029 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734013 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 22:47:32.741029 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734017 2578 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 22:47:32.741029 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734022 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 22:47:32.741625 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734027 2578 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 22:47:32.741625 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734032 2578 flags.go:64] FLAG: --kube-reserved="" Nov 02 22:47:32.741625 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734037 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 22:47:32.741625 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734042 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 22:47:32.741625 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734047 2578 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 22:47:32.741625 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734052 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 22:47:32.741625 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734056 2578 flags.go:64] FLAG: --lock-file="" Nov 02 22:47:32.741625 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734061 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 22:47:32.741625 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734066 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 22:47:32.741625 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734072 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 22:47:32.741625 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734084 2578 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 22:47:32.741625 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734088 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 22:47:32.741625 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734093 2578 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 22:47:32.741625 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734098 2578 flags.go:64] FLAG: --logging-format="text" Nov 02 22:47:32.741625 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734103 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 22:47:32.741625 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734108 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 22:47:32.741625 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734112 2578 flags.go:64] FLAG: --manifest-url="" Nov 02 22:47:32.741625 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734117 2578 flags.go:64] FLAG: --manifest-url-header="" Nov 02 22:47:32.741625 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734125 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 22:47:32.741625 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734130 2578 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 22:47:32.741625 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734137 2578 flags.go:64] FLAG: --max-pods="110" Nov 02 22:47:32.741625 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734142 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 22:47:32.741625 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734147 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 22:47:32.741625 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734152 2578 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 22:47:32.741625 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734157 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734162 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734167 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734172 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734184 2578 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734189 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734194 2578 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734199 2578 flags.go:64] FLAG: --pod-cidr="" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734204 2578 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734209 2578 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734214 2578 flags.go:64] FLAG: --pods-per-core="0" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734218 2578 flags.go:64] FLAG: --port="10250" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734223 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734228 2578 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0900d3cfc04b8d8b0" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734233 2578 flags.go:64] FLAG: --qos-reserved="" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734238 2578 flags.go:64] FLAG: --read-only-port="10255" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734243 2578 flags.go:64] FLAG: --register-node="true" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734248 2578 flags.go:64] FLAG: --register-with-taints="" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734257 2578 flags.go:64] FLAG: --registry-burst="10" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734262 2578 flags.go:64] FLAG: --registry-qps="5" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734266 2578 flags.go:64] FLAG: --reserved-cpus="" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734271 2578 flags.go:64] FLAG: --reserved-memory="" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734278 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734283 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734288 2578 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734293 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 22:47:32.742298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734298 2578 flags.go:64] FLAG: --runonce="false" Nov 02 22:47:32.743068 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734303 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 22:47:32.743068 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734307 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 22:47:32.743068 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734312 2578 flags.go:64] FLAG: --seccomp-default="false" Nov 02 22:47:32.743068 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734317 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 22:47:32.743068 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734322 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 22:47:32.743068 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734327 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 22:47:32.743068 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734332 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 22:47:32.743068 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734336 2578 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 22:47:32.743068 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734341 2578 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 22:47:32.743068 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734346 2578 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 22:47:32.743068 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734351 2578 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 22:47:32.743068 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734355 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 22:47:32.743068 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734360 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 22:47:32.743068 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734366 2578 flags.go:64] FLAG: --system-cgroups="" Nov 02 22:47:32.743068 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734371 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 22:47:32.743068 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734379 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 22:47:32.743068 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734384 2578 flags.go:64] FLAG: --tls-cert-file="" Nov 02 22:47:32.743068 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734388 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 22:47:32.743068 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734395 2578 flags.go:64] FLAG: --tls-min-version="" Nov 02 22:47:32.743068 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734399 2578 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 22:47:32.743068 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734404 2578 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 22:47:32.743068 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734409 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 22:47:32.743068 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734414 2578 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 22:47:32.743068 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734420 2578 flags.go:64] FLAG: --v="2" Nov 02 22:47:32.743068 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734428 2578 flags.go:64] FLAG: --version="false" Nov 02 22:47:32.743681 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734435 2578 flags.go:64] FLAG: --vmodule="" Nov 02 22:47:32.743681 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734441 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 22:47:32.743681 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.734448 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 22:47:32.743681 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734701 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:47:32.743681 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734709 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:47:32.743681 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734714 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:47:32.743681 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734719 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:47:32.743681 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734723 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:47:32.743681 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734728 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:47:32.743681 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734732 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:47:32.743681 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734736 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:47:32.743681 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734741 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:47:32.743681 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734745 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:47:32.743681 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734749 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:47:32.743681 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734753 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:47:32.743681 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734758 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:47:32.743681 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734766 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:47:32.743681 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734771 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:47:32.743681 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734775 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:47:32.743681 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734779 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:47:32.744156 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734784 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:47:32.744156 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734788 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:47:32.744156 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734792 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:47:32.744156 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734796 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:47:32.744156 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734800 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:47:32.744156 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734803 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:47:32.744156 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734807 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:47:32.744156 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734811 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:47:32.744156 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734815 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:47:32.744156 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734819 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:47:32.744156 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734825 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:47:32.744156 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734831 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:47:32.744156 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734835 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:47:32.744156 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734840 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:47:32.744156 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734846 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:47:32.744156 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734850 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:47:32.744156 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734854 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:47:32.744156 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734859 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:47:32.744156 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734863 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:47:32.744156 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734867 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:47:32.744657 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734872 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:47:32.744657 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734876 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:47:32.744657 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734881 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:47:32.744657 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734885 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:47:32.744657 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734889 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:47:32.744657 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734893 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:47:32.744657 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734898 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:47:32.744657 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734902 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:47:32.744657 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734906 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:47:32.744657 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734911 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:47:32.744657 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734916 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:47:32.744657 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734920 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:47:32.744657 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734925 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:47:32.744657 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734929 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:47:32.744657 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734934 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:47:32.744657 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734938 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:47:32.744657 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734942 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:47:32.744657 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734946 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:47:32.744657 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734950 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:47:32.745114 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734954 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:47:32.745114 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734958 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:47:32.745114 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734963 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:47:32.745114 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734968 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:47:32.745114 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734972 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:47:32.745114 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734977 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:47:32.745114 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734981 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:47:32.745114 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734987 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:47:32.745114 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734992 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:47:32.745114 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.734995 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:47:32.745114 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735000 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:47:32.745114 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735004 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:47:32.745114 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735008 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:47:32.745114 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735013 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:47:32.745114 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735017 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:47:32.745114 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735021 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:47:32.745114 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735025 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:47:32.745114 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735030 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:47:32.745114 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735034 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:47:32.745626 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735040 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:47:32.745626 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735047 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:47:32.745626 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735051 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:47:32.745626 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735056 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:47:32.745626 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735061 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:47:32.745626 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735068 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:47:32.745626 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735073 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:47:32.745626 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735078 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:47:32.745626 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735082 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:47:32.745626 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735087 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:47:32.745626 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735091 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:47:32.745626 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735095 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:47:32.745626 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735099 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:47:32.745626 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735104 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:47:32.745626 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735108 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:47:32.745626 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735112 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:47:32.745626 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735119 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:47:32.745626 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735123 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:47:32.745626 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735128 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:47:32.746091 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735132 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:47:32.746091 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735138 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:47:32.746091 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735142 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:47:32.746091 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735146 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:47:32.746091 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735151 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:47:32.746091 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735155 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:47:32.746091 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735159 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:47:32.746091 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735163 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:47:32.746091 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735167 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:47:32.746091 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735171 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:47:32.746091 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735175 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:47:32.746091 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735179 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:47:32.746091 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.735226 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:47:32.746091 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735292 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:47:32.746091 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735299 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:47:32.746091 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735304 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:47:32.746498 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735309 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:47:32.746498 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735314 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:47:32.746498 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735319 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:47:32.746498 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735324 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:47:32.746498 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735329 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:47:32.746498 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735333 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:47:32.746498 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735338 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:47:32.746498 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735343 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:47:32.746498 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735347 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:47:32.746498 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735351 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:47:32.746498 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735356 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:47:32.746498 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735360 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:47:32.746498 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735364 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:47:32.746498 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735369 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:47:32.746498 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735375 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:47:32.746498 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735379 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:47:32.746498 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735384 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:47:32.746498 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735390 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:47:32.746498 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735394 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:47:32.746498 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735399 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:47:32.746978 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735403 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:47:32.746978 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735407 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:47:32.746978 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735412 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:47:32.746978 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735416 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:47:32.746978 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735420 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:47:32.746978 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735424 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:47:32.746978 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735428 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:47:32.746978 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735433 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:47:32.746978 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735437 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:47:32.746978 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735441 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:47:32.746978 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735446 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:47:32.746978 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735450 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:47:32.746978 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735454 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:47:32.746978 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735461 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:47:32.746978 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735468 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:47:32.746978 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735473 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:47:32.746978 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735496 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:47:32.746978 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735502 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:47:32.746978 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735508 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:47:32.747670 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735514 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:47:32.747670 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735519 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:47:32.747670 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735524 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:47:32.747670 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735528 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:47:32.747670 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735532 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:47:32.747670 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735537 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:47:32.747670 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735541 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:47:32.747670 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735547 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:47:32.747670 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735551 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:47:32.747670 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735555 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:47:32.747670 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735561 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:47:32.747670 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735566 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:47:32.747670 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735571 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:47:32.747670 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735575 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:47:32.747670 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735579 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:47:32.747670 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735584 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:47:32.747670 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735588 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:47:32.747670 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735592 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:47:32.747670 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735597 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:47:32.747670 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735602 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:47:32.748359 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735606 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:47:32.748359 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735611 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:47:32.748359 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735615 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:47:32.748359 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735619 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:47:32.748359 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735623 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:47:32.748359 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735627 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:47:32.748359 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735631 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:47:32.748359 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735636 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:47:32.748359 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735640 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:47:32.748359 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735644 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:47:32.748359 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735648 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:47:32.748359 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735652 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:47:32.748359 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735656 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:47:32.748359 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735660 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:47:32.748359 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735665 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:47:32.748359 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735669 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:47:32.748359 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735674 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:47:32.748359 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735678 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:47:32.748359 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735682 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:47:32.748359 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735688 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:47:32.748866 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735692 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:47:32.748866 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735697 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:47:32.748866 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735702 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:47:32.748866 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735706 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:47:32.748866 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735711 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:47:32.748866 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735715 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:47:32.748866 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735719 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:47:32.748866 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735724 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:47:32.748866 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735728 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:47:32.748866 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735732 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:47:32.748866 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735736 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:47:32.748866 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735741 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:47:32.748866 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735745 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:47:32.748866 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735749 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:47:32.748866 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735754 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:47:32.748866 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735758 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:47:32.748866 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735763 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:47:32.748866 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735767 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:47:32.748866 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735771 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:47:32.748866 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735777 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:47:32.749338 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735782 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:47:32.749338 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735786 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:47:32.749338 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735791 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:47:32.749338 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.735795 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:47:32.749338 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.743646 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 22:47:32.749338 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.743664 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 22:47:32.749338 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.743676 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 22:47:32.749338 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.743680 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 22:47:32.749338 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743739 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:47:32.749338 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743744 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:47:32.749338 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743748 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:47:32.749338 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743751 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:47:32.749338 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743755 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:47:32.749338 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743757 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:47:32.749338 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743761 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:47:32.749338 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743764 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:47:32.749338 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743767 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:47:32.749338 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743770 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:47:32.749338 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743772 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:47:32.749338 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743775 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:47:32.749955 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743778 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:47:32.749955 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743780 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:47:32.749955 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743783 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:47:32.749955 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743786 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:47:32.749955 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743789 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:47:32.749955 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743791 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:47:32.749955 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743794 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:47:32.749955 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743797 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:47:32.749955 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743799 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:47:32.749955 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743803 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:47:32.749955 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743807 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:47:32.749955 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743810 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:47:32.749955 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743814 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:47:32.749955 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743817 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:47:32.749955 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743819 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:47:32.749955 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743822 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:47:32.749955 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743824 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:47:32.749955 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743828 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:47:32.749955 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743831 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:47:32.750407 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743834 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:47:32.750407 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743836 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:47:32.750407 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743839 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:47:32.750407 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743842 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:47:32.750407 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743844 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:47:32.750407 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743847 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:47:32.750407 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743849 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:47:32.750407 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743852 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:47:32.750407 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743854 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:47:32.750407 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743857 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:47:32.750407 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743859 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:47:32.750407 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743862 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:47:32.750407 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743864 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:47:32.750407 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743867 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:47:32.750407 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743869 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:47:32.750407 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743872 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:47:32.750407 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743875 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:47:32.750407 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743877 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:47:32.750407 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743880 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:47:32.750407 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743883 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:47:32.750905 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743885 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:47:32.750905 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743888 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:47:32.750905 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743890 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:47:32.750905 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743893 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:47:32.750905 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743895 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:47:32.750905 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743898 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:47:32.750905 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743901 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:47:32.750905 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743904 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:47:32.750905 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743906 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:47:32.750905 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743909 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:47:32.750905 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743912 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:47:32.750905 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743915 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:47:32.750905 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743918 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:47:32.750905 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743920 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:47:32.750905 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743923 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:47:32.750905 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743926 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:47:32.750905 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743928 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:47:32.750905 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743931 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:47:32.750905 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743933 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:47:32.750905 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743936 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:47:32.751403 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743938 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:47:32.751403 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743941 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:47:32.751403 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743944 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:47:32.751403 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743946 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:47:32.751403 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743949 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:47:32.751403 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743951 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:47:32.751403 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743954 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:47:32.751403 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743956 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:47:32.751403 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743959 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:47:32.751403 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743961 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:47:32.751403 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743963 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:47:32.751403 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743966 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:47:32.751403 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743968 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:47:32.751403 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743971 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:47:32.751403 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743973 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:47:32.751403 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743976 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:47:32.751403 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743978 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:47:32.751403 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743981 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:47:32.751403 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743984 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:47:32.751403 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743986 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:47:32.751892 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743989 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:47:32.751892 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743991 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:47:32.751892 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743994 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:47:32.751892 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.743997 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:47:32.751892 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744000 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:47:32.751892 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744003 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:47:32.751892 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744005 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:47:32.751892 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744008 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:47:32.751892 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744010 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:47:32.751892 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744013 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:47:32.751892 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744015 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:47:32.751892 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744019 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:47:32.751892 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744023 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:47:32.751892 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744025 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:47:32.751892 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744028 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:47:32.751892 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.744059 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:47:32.752270 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744201 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:47:32.752270 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744205 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:47:32.752270 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744208 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:47:32.752270 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744210 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:47:32.752270 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744213 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:47:32.752270 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744215 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:47:32.752270 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744218 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:47:32.752270 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744220 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:47:32.752270 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744223 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:47:32.752270 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744225 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:47:32.752270 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744228 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:47:32.752270 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744230 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:47:32.752270 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744233 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:47:32.752270 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744235 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:47:32.752270 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744238 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:47:32.752270 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744241 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:47:32.752270 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744243 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:47:32.752270 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744246 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:47:32.752270 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744249 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:47:32.752270 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744252 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:47:32.752809 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744254 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:47:32.752809 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744257 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:47:32.752809 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744260 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:47:32.752809 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744262 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:47:32.752809 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744265 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:47:32.752809 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744267 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:47:32.752809 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744269 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:47:32.752809 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744272 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:47:32.752809 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744274 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:47:32.752809 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744277 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:47:32.752809 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744279 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:47:32.752809 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744281 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:47:32.752809 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744284 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:47:32.752809 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744286 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:47:32.752809 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744289 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:47:32.752809 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744291 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:47:32.752809 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744294 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:47:32.752809 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744297 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:47:32.752809 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744299 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:47:32.752809 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744302 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:47:32.753278 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744304 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:47:32.753278 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744307 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:47:32.753278 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744309 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:47:32.753278 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744312 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:47:32.753278 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744314 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:47:32.753278 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744317 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:47:32.753278 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744320 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:47:32.753278 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744322 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:47:32.753278 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744325 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:47:32.753278 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744327 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:47:32.753278 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744330 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:47:32.753278 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744333 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:47:32.753278 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744336 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:47:32.753278 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744338 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:47:32.753278 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744341 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:47:32.753278 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744344 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:47:32.753278 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744346 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:47:32.753278 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744348 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:47:32.753278 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744352 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:47:32.753811 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744356 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:47:32.753811 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744359 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:47:32.753811 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744363 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:47:32.753811 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744365 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:47:32.753811 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744368 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:47:32.753811 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744370 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:47:32.753811 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744372 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:47:32.753811 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744375 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:47:32.753811 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744377 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:47:32.753811 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744380 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:47:32.753811 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744382 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:47:32.753811 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744385 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:47:32.753811 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744387 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:47:32.753811 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744390 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:47:32.753811 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744392 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:47:32.753811 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744395 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:47:32.753811 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744397 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:47:32.753811 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744400 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:47:32.753811 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744402 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:47:32.753811 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744405 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:47:32.754291 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744407 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:47:32.754291 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744410 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:47:32.754291 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744412 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:47:32.754291 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744415 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:47:32.754291 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744418 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:47:32.754291 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744420 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:47:32.754291 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744423 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:47:32.754291 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744425 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:47:32.754291 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744428 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:47:32.754291 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744430 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:47:32.754291 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744433 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:47:32.754291 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744435 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:47:32.754291 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744438 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:47:32.754291 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744440 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:47:32.754291 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744443 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:47:32.754291 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744445 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:47:32.754291 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744448 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:47:32.754291 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744450 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:47:32.754291 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744453 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:47:32.754784 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744455 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:47:32.754784 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744458 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:47:32.754784 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744460 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:47:32.754784 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744463 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:47:32.754784 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744465 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:47:32.754784 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744468 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:47:32.754784 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744472 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:47:32.754784 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744475 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:47:32.754784 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.744516 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:47:32.754784 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744549 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:47:32.754784 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744553 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:47:32.754784 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744556 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:47:32.754784 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744558 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:47:32.754784 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744561 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:47:32.754784 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744564 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:47:32.755154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744566 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:47:32.755154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744569 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:47:32.755154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744571 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:47:32.755154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744574 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:47:32.755154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744576 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:47:32.755154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744579 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:47:32.755154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744582 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:47:32.755154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744585 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:47:32.755154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744587 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:47:32.755154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744590 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:47:32.755154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744592 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:47:32.755154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744595 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:47:32.755154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744597 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:47:32.755154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744600 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:47:32.755154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744602 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:47:32.755154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744604 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:47:32.755154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744607 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:47:32.755154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744609 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:47:32.755154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744612 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:47:32.755690 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744614 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:47:32.755690 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744618 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:47:32.755690 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744621 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:47:32.755690 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744624 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:47:32.755690 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744626 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:47:32.755690 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744629 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:47:32.755690 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744632 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:47:32.755690 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744635 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:47:32.755690 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744637 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:47:32.755690 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744640 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:47:32.755690 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744642 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:47:32.755690 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744644 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:47:32.755690 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744647 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:47:32.755690 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744650 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:47:32.755690 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744652 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:47:32.755690 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744655 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:47:32.755690 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744657 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:47:32.755690 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744660 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:47:32.755690 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744663 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:47:32.755690 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744665 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:47:32.756184 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744668 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:47:32.756184 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744671 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:47:32.756184 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744673 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:47:32.756184 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744676 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:47:32.756184 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744678 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:47:32.756184 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744681 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:47:32.756184 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744683 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:47:32.756184 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744686 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:47:32.756184 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744689 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:47:32.756184 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744691 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:47:32.756184 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744694 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:47:32.756184 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744696 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:47:32.756184 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744698 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:47:32.756184 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744701 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:47:32.756184 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744704 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:47:32.756184 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744706 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:47:32.756184 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744709 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:47:32.756184 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744711 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:47:32.756184 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744714 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:47:32.756697 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744716 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:47:32.756697 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744719 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:47:32.756697 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744721 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:47:32.756697 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744724 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:47:32.756697 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744726 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:47:32.756697 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744729 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:47:32.756697 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744732 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:47:32.756697 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744734 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:47:32.756697 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744738 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:47:32.756697 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744742 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:47:32.756697 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744744 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:47:32.756697 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744747 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:47:32.756697 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744750 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:47:32.756697 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744752 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:47:32.756697 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744754 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:47:32.756697 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744757 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:47:32.756697 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744760 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:47:32.756697 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744763 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:47:32.756697 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744765 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:47:32.757154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744767 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:47:32.757154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744770 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:47:32.757154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744772 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:47:32.757154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744774 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:47:32.757154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744777 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:47:32.757154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744779 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:47:32.757154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744782 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:47:32.757154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744784 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:47:32.757154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744787 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:47:32.757154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744790 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:47:32.757154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744793 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:47:32.757154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744796 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:47:32.757154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744798 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:47:32.757154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744800 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:47:32.757154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744802 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:47:32.757154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744805 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:47:32.757154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744807 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:47:32.757154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744809 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:47:32.757154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744812 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:47:32.757154 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744815 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:47:32.757659 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744817 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:47:32.757659 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744820 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:47:32.757659 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:32.744822 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:47:32.757659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.744904 2578 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 22:47:32.757659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.749531 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 22:47:32.757659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.750536 2578 server.go:1014] "Starting client certificate rotation" Nov 02 22:47:32.757659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.750629 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 22:47:32.757659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.750663 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 22:47:32.757659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.752686 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:47:32.757659 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.753876 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:47:32.766060 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.766040 2578 log.go:25] "Validated CRI v1 runtime API" Nov 02 22:47:32.770073 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.770055 2578 log.go:25] "Validated CRI v1 image API" Nov 02 22:47:32.770968 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.770940 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 22:47:32.773752 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.773719 2578 fs.go:135] Filesystem UUIDs: map[673ecb43-8824-46ab-b3a9-31f332ba0903:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 86fde7a3-20a9-464e-bd82-e9a95e267ec3:/dev/nvme0n1p4] Nov 02 22:47:32.773854 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.773749 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 22:47:32.778327 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.778192 2578 manager.go:217] Machine: {Timestamp:2026-11-02 22:47:32.777676931 +0000 UTC m=+0.304321547 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3093834 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21efe55406fefe399efc598815a689 SystemUUID:ec21efe5-5406-fefe-399e-fc598815a689 BootID:2dab55aa-a2ba-4c2d-851e-5da4b2003631 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:47:1b:b2:82:95 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:47:1b:b2:82:95 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:d4:93:73:8d:b2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 22:47:32.778327 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.778310 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 22:47:32.778516 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.778446 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 22:47:32.781544 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.781516 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 22:47:32.781702 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.781546 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-250.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 22:47:32.781747 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.781705 2578 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 22:47:32.781747 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.781715 2578 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 22:47:32.781747 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.781739 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:47:32.782664 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.782652 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:47:32.783801 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.783790 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 22:47:32.788194 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.788170 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 22:47:32.788346 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.788336 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 22:47:32.789151 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.789140 2578 kubelet.go:493] "Attempting to sync node with API server" Nov 02 22:47:32.789190 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.789155 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 22:47:32.789190 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.789167 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 22:47:32.789190 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.789177 2578 kubelet.go:405] "Adding apiserver pod source" Nov 02 22:47:32.789190 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.789186 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 22:47:32.789908 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.789887 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:47:32.792607 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.792577 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 22:47:32.800051 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.800031 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:47:32.800941 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.800928 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 22:47:32.800977 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.800966 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 22:47:32.802387 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.802375 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 22:47:32.802429 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.802393 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 22:47:32.802429 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.802400 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 22:47:32.802429 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.802407 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 22:47:32.802429 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.802413 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 22:47:32.802429 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.802419 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 22:47:32.803120 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.803110 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 22:47:32.803120 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.803121 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 22:47:32.803182 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.803130 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 22:47:32.803182 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.803136 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 22:47:32.803182 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.803145 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 22:47:32.803182 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.803154 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 22:47:32.804656 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.804643 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 22:47:32.804656 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.804655 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 22:47:32.809131 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.809116 2578 server.go:1263] "Started kubelet" Nov 02 22:47:32.809266 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.809231 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 22:47:32.809330 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.809235 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 22:47:32.809330 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.809317 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 22:47:32.809902 ip-10-0-135-250 systemd[1]: Started Kubernetes Kubelet. Nov 02 22:47:32.810699 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.810679 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 22:47:32.812252 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.812233 2578 server.go:317] "Adding debug handlers to kubelet server" Nov 02 22:47:32.814113 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:32.814092 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 22:47:32.818427 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.818403 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 22:47:32.818582 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.818409 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 22:47:32.818582 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:32.818556 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-250.us-west-1.compute.internal\" not found" Nov 02 22:47:32.819392 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.819326 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 22:47:32.819596 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.819547 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 22:47:32.819596 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.819581 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 22:47:32.819784 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.819683 2578 factory.go:55] Registering systemd factory Nov 02 22:47:32.819784 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.819701 2578 factory.go:223] Registration of the systemd container factory successfully Nov 02 22:47:32.819784 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.819722 2578 reconstruct.go:97] "Volume reconstruction finished" Nov 02 22:47:32.819784 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.819749 2578 reconciler.go:29] "Reconciler: start to sync state" Nov 02 22:47:32.819966 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.819904 2578 factory.go:153] Registering CRI-O factory Nov 02 22:47:32.819966 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.819921 2578 factory.go:223] Registration of the crio container factory successfully Nov 02 22:47:32.820062 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.819982 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 22:47:32.820062 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.820002 2578 factory.go:103] Registering Raw factory Nov 02 22:47:32.820062 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.820030 2578 manager.go:1199] Started watching for new ooms in manager Nov 02 22:47:32.821137 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.821120 2578 manager.go:319] Starting recovery of all containers Nov 02 22:47:32.828101 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.828035 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 22:47:32.830111 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.830090 2578 manager.go:324] Recovery completed Nov 02 22:47:32.832156 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:32.832129 2578 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Nov 02 22:47:32.835003 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.834987 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:47:32.837230 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.837207 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:47:32.837385 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.837373 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:47:32.837472 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.837463 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:47:32.838564 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.838548 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:47:32.838651 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.838583 2578 cpu_manager.go:226] "Starting" policy="none" Nov 02 22:47:32.838651 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.838594 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 22:47:32.838651 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.838619 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 22:47:32.839351 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.839335 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:47:32.839381 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.839373 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:47:32.839411 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.839387 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:47:32.840747 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.840734 2578 policy_none.go:50] "Start" Nov 02 22:47:32.840781 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.840753 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 22:47:32.840781 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.840763 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 22:47:32.842796 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.842785 2578 policy_none.go:44] "Start" Nov 02 22:47:32.892382 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.892289 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 22:47:32.895389 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.895373 2578 manager.go:341] "Starting Device Plugin manager" Nov 02 22:47:32.895458 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:32.895413 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 22:47:32.895458 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.895428 2578 server.go:87] "Starting device plugin registration server" Nov 02 22:47:32.895815 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.895801 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 22:47:32.895873 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.895819 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 22:47:32.895974 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.895951 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 22:47:32.896076 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.896065 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 22:47:32.896076 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.896074 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 22:47:32.896752 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:32.896726 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 22:47:32.896842 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:32.896803 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-250.us-west-1.compute.internal\" not found" Nov 02 22:47:32.950130 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.949957 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-250.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 22:47:32.953121 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:32.949963 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-250.us-west-1.compute.internal.18e45bf3759fc5b4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-250.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-250.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 22:47:32.808885684 +0000 UTC m=+0.335530300,LastTimestamp:2026-11-02 22:47:32.808885684 +0000 UTC m=+0.335530300,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-250.us-west-1.compute.internal,}" Nov 02 22:47:32.970943 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.970915 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 22:47:32.971064 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.970948 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 22:47:32.971064 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.970977 2578 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 22:47:32.971064 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:32.971018 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 22:47:32.996803 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.996690 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:47:32.997776 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.997757 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:47:32.997900 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.997795 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:47:32.997900 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.997808 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:47:32.997900 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:32.997842 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:47:32.998093 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:32.998071 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-250.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 22:47:33.029837 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.029804 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 22:47:33.029991 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.029884 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zwbq2" Nov 02 22:47:33.077105 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.077063 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal"] Nov 02 22:47:33.077270 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.077140 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:47:33.078245 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.078229 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:47:33.078344 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.078258 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:47:33.078344 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.078270 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:47:33.080623 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.080609 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:47:33.080768 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.080752 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:47:33.080815 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.080787 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:47:33.081376 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.081361 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:47:33.081441 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.081388 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:47:33.081441 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.081398 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:47:33.081533 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.081365 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:47:33.081533 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.081460 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:47:33.081533 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.081474 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:47:33.083527 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.083512 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:47:33.083602 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.083538 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:47:33.084220 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.084202 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:47:33.084283 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.084237 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:47:33.084283 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.084265 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:47:33.108697 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.108668 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:47:33.109975 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.109957 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:47:33.114714 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:33.114695 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-250.us-west-1.compute.internal\" not found" node="ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:47:33.117400 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:33.117383 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-250.us-west-1.compute.internal\" not found" node="ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:47:33.120654 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.120633 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c6e8e95c46679d654a23b98c776960b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal\" (UID: \"7c6e8e95c46679d654a23b98c776960b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:47:33.120734 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.120666 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c6e8e95c46679d654a23b98c776960b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal\" (UID: \"7c6e8e95c46679d654a23b98c776960b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:47:33.120734 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.120691 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09bea7df1640e9a4e82191575067a028-config\") pod \"kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal\" (UID: \"09bea7df1640e9a4e82191575067a028\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:47:33.208609 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.208577 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:47:33.208720 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.208655 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:47:33.209801 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.209779 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:47:33.209881 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:33.209808 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-250.us-west-1.compute.internal\": node \"ip-10-0-135-250.us-west-1.compute.internal\" not found" Nov 02 22:47:33.219136 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.219111 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:47:33.221329 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.221310 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09bea7df1640e9a4e82191575067a028-config\") pod \"kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal\" (UID: \"09bea7df1640e9a4e82191575067a028\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:47:33.221410 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.221344 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c6e8e95c46679d654a23b98c776960b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal\" (UID: \"7c6e8e95c46679d654a23b98c776960b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:47:33.221410 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.221369 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c6e8e95c46679d654a23b98c776960b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal\" (UID: \"7c6e8e95c46679d654a23b98c776960b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:47:33.221529 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.221441 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c6e8e95c46679d654a23b98c776960b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal\" (UID: \"7c6e8e95c46679d654a23b98c776960b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:47:33.221529 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.221444 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09bea7df1640e9a4e82191575067a028-config\") pod \"kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal\" (UID: \"09bea7df1640e9a4e82191575067a028\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:47:33.221529 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.221444 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c6e8e95c46679d654a23b98c776960b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal\" (UID: \"7c6e8e95c46679d654a23b98c776960b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:47:33.279660 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.279590 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:47:33.279787 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.279709 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:47:33.336993 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.336964 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:47:33.417933 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.417902 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:47:33.420061 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.420041 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:47:33.750183 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.750094 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 22:47:33.750788 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.750335 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 22:47:33.750788 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.750335 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 22:47:33.750788 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.750335 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 22:47:33.750788 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:33.750383 2578 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ae6c114d507d041eea76c03a1ba35e05-cac881102500eba0.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-135-250.us-west-1.compute.internal.18e45bf37755279c\": read tcp 10.0.135.250:42348->77.113.138.105:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-135-250.us-west-1.compute.internal.18e45bf37755279c default 5269 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-250.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-250.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-250.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 22:47:32 +0000 UTC,LastTimestamp:2026-11-02 22:47:32.997813303 +0000 UTC m=+0.524457922,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-250.us-west-1.compute.internal,}" Nov 02 22:47:33.790337 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.790299 2578 apiserver.go:52] "Watching apiserver" Nov 02 22:47:33.819298 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:33.819266 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 22:47:34.034229 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.034125 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 22:42:33 +0000 UTC" deadline="2028-04-23 14:14:25.298158742 +0000 UTC" Nov 02 22:47:34.034229 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.034167 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12903h26m51.263994619s" Nov 02 22:47:34.058501 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.058451 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 22:47:34.058816 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.058797 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5","openshift-image-registry/node-ca-dsp96","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal","kube-system/konnectivity-agent-ftp5v","openshift-cluster-node-tuning-operator/tuned-9pf69","openshift-dns/node-resolver-l5tth","openshift-multus/multus-additional-cni-plugins-lpqcq","openshift-multus/multus-hg78p","openshift-multus/network-metrics-daemon-fqwl4","openshift-network-diagnostics/network-check-target-569mb","openshift-network-operator/iptables-alerter-jvpc5","openshift-ovn-kubernetes/ovnkube-node-d25g6","kube-system/global-pull-secret-syncer-9drjv"] Nov 02 22:47:34.061313 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.061292 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:47:34.063307 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.063274 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dsp96" Nov 02 22:47:34.065312 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.065286 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ftp5v" Nov 02 22:47:34.067445 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.067425 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.069505 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.069492 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l5tth" Nov 02 22:47:34.071663 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.071644 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:47:34.074294 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.074275 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.077147 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.077133 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:47:34.077205 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:34.077191 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqwl4" podUID="5ce08d97-9151-4a82-92b3-a01a4057f046" Nov 02 22:47:34.079203 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.079188 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:47:34.079251 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:34.079240 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-569mb" podUID="f8acfea0-a4d3-4673-9fc6-e48a94dfbba4" Nov 02 22:47:34.081280 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.081268 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jvpc5" Nov 02 22:47:34.083539 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.083518 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.085981 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.085958 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:34.086088 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:34.086026 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9drjv" podUID="e608a3bc-a7f2-45a1-9603-d85bbbe20614" Nov 02 22:47:34.113140 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.113120 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 22:47:34.113226 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.113158 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 22:47:34.113265 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.113234 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 22:47:34.113338 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.113326 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bxrdd\"" Nov 02 22:47:34.114401 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.114390 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 22:47:34.114863 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.114848 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 22:47:34.114926 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.114916 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 22:47:34.114993 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.114983 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d58gp\"" Nov 02 22:47:34.116455 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.116438 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 22:47:34.116864 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.116850 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 22:47:34.116911 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.116895 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pl59x\"" Nov 02 22:47:34.118560 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.118548 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:47:34.118846 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.118835 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 22:47:34.118988 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.118977 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p2dkz\"" Nov 02 22:47:34.120592 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.120573 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f4f84\"" Nov 02 22:47:34.120801 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.120787 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 22:47:34.120861 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.120829 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 22:47:34.120895 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.120861 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 22:47:34.122903 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.122889 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 22:47:34.125186 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125171 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-sysconfig\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.125252 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125194 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1062888a-e10c-4a34-b84a-f85ff9eb8063-agent-certs\") pod \"konnectivity-agent-ftp5v\" (UID: \"1062888a-e10c-4a34-b84a-f85ff9eb8063\") " pod="kube-system/konnectivity-agent-ftp5v" Nov 02 22:47:34.125252 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125212 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-var-lib-cni-bin\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.125252 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125226 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs\") pod \"network-metrics-daemon-fqwl4\" (UID: \"5ce08d97-9151-4a82-92b3-a01a4057f046\") " pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:47:34.125252 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125244 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxxkn\" (UniqueName: \"kubernetes.io/projected/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4-kube-api-access-kxxkn\") pod \"network-check-target-569mb\" (UID: \"f8acfea0-a4d3-4673-9fc6-e48a94dfbba4\") " pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:47:34.125415 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125263 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0e82120d-ed55-489b-a3d5-a553ac1b1678-env-overrides\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.125415 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125308 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-systemd\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.125415 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125344 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-var-lib-cni-multus\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.125415 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125378 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-multus-conf-dir\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.125554 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125409 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-run-multus-certs\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.125554 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125473 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32856eec-17a2-4d3a-8ca1-53755398f81c-host-slash\") pod \"iptables-alerter-jvpc5\" (UID: \"32856eec-17a2-4d3a-8ca1-53755398f81c\") " pod="openshift-network-operator/iptables-alerter-jvpc5" Nov 02 22:47:34.125554 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125525 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-var-lib-kubelet\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.125554 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125550 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-var-lib-openvswitch\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.125667 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125566 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-sysctl-d\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.125667 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125594 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:47:34.125667 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125620 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lppg\" (UniqueName: \"kubernetes.io/projected/fabb0266-226c-4935-a9fc-5ed6199e2f9b-kube-api-access-9lppg\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:47:34.125667 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125644 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a5e437f-4589-4f76-b777-026186f06c73-cni-binary-copy\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.125788 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125681 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-etc-openvswitch\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.125788 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125708 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73682abc-fb01-4ac2-a073-fcffad51a512-host\") pod \"node-ca-dsp96\" (UID: \"73682abc-fb01-4ac2-a073-fcffad51a512\") " pod="openshift-image-registry/node-ca-dsp96" Nov 02 22:47:34.125788 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125723 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-host\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.125788 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125763 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm5cb\" (UniqueName: \"kubernetes.io/projected/d05df5bf-8c7c-4c79-afa6-96fc7d3689e4-kube-api-access-cm5cb\") pod \"node-resolver-l5tth\" (UID: \"d05df5bf-8c7c-4c79-afa6-96fc7d3689e4\") " pod="openshift-dns/node-resolver-l5tth" Nov 02 22:47:34.125900 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125794 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcgff\" (UniqueName: \"kubernetes.io/projected/ca56f427-3734-48ec-b7ce-100daeb7b3bb-kube-api-access-pcgff\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:47:34.125900 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125824 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca56f427-3734-48ec-b7ce-100daeb7b3bb-cnibin\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:47:34.125900 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125840 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-run-ovn-kubernetes\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.125900 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125875 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-sys\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.126010 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125896 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-sys-fs\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:47:34.126010 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125924 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-system-cni-dir\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.126010 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125945 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a5e437f-4589-4f76-b777-026186f06c73-multus-daemon-config\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.126010 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.125978 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e608a3bc-a7f2-45a1-9603-d85bbbe20614-kubelet-config\") pod \"global-pull-secret-syncer-9drjv\" (UID: \"e608a3bc-a7f2-45a1-9603-d85bbbe20614\") " pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:34.126114 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.126010 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-kubernetes\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.126114 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.126035 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca56f427-3734-48ec-b7ce-100daeb7b3bb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:47:34.126114 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.126053 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-socket-dir\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:47:34.126114 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.126071 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpfqn\" (UniqueName: \"kubernetes.io/projected/5ce08d97-9151-4a82-92b3-a01a4057f046-kube-api-access-qpfqn\") pod \"network-metrics-daemon-fqwl4\" (UID: \"5ce08d97-9151-4a82-92b3-a01a4057f046\") " pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:47:34.126261 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.126121 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0e82120d-ed55-489b-a3d5-a553ac1b1678-ovnkube-config\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.126261 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.126146 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj75b\" (UniqueName: \"kubernetes.io/projected/0e82120d-ed55-489b-a3d5-a553ac1b1678-kube-api-access-sj75b\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.126261 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.126162 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnvrv\" (UniqueName: \"kubernetes.io/projected/73682abc-fb01-4ac2-a073-fcffad51a512-kube-api-access-nnvrv\") pod \"node-ca-dsp96\" (UID: \"73682abc-fb01-4ac2-a073-fcffad51a512\") " pod="openshift-image-registry/node-ca-dsp96" Nov 02 22:47:34.126261 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.126186 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-run\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.126261 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.126213 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-lib-modules\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.126261 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.126234 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32856eec-17a2-4d3a-8ca1-53755398f81c-iptables-alerter-script\") pod \"iptables-alerter-jvpc5\" (UID: \"32856eec-17a2-4d3a-8ca1-53755398f81c\") " pod="openshift-network-operator/iptables-alerter-jvpc5" Nov 02 22:47:34.126425 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.126267 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-slash\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.126425 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.126291 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-run-openvswitch\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.126425 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.126314 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-node-log\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.126425 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.126370 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-cni-netd\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.126425 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.126393 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e82120d-ed55-489b-a3d5-a553ac1b1678-ovn-node-metrics-cert\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.126425 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.126408 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee507e21-156a-48a5-9c73-5f9e98bdd44a-tmp\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.126630 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.126461 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca56f427-3734-48ec-b7ce-100daeb7b3bb-cni-binary-copy\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:47:34.126630 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.126499 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca56f427-3734-48ec-b7ce-100daeb7b3bb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:47:34.126630 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.126523 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-systemd-units\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.126630 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.126538 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0e82120d-ed55-489b-a3d5-a553ac1b1678-ovnkube-script-lib\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.126630 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.126553 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-sysctl-conf\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.126630 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.126575 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j2w5\" (UniqueName: \"kubernetes.io/projected/ee507e21-156a-48a5-9c73-5f9e98bdd44a-kube-api-access-6j2w5\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.127032 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.127011 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca56f427-3734-48ec-b7ce-100daeb7b3bb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:47:34.127096 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.127075 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-registration-dir\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:47:34.127151 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.127114 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-run-netns\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.127189 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.127164 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-hostroot\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.127241 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.127185 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e608a3bc-a7f2-45a1-9603-d85bbbe20614-dbus\") pod \"global-pull-secret-syncer-9drjv\" (UID: \"e608a3bc-a7f2-45a1-9603-d85bbbe20614\") " pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:34.129297 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.127553 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d05df5bf-8c7c-4c79-afa6-96fc7d3689e4-hosts-file\") pod \"node-resolver-l5tth\" (UID: \"d05df5bf-8c7c-4c79-afa6-96fc7d3689e4\") " pod="openshift-dns/node-resolver-l5tth" Nov 02 22:47:34.129297 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.127602 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca56f427-3734-48ec-b7ce-100daeb7b3bb-system-cni-dir\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:47:34.129297 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.127692 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-device-dir\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:47:34.129297 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.127731 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-multus-cni-dir\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.129297 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.127807 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66b2k\" (UniqueName: \"kubernetes.io/projected/0a5e437f-4589-4f76-b777-026186f06c73-kube-api-access-66b2k\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.129297 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.127839 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/73682abc-fb01-4ac2-a073-fcffad51a512-serviceca\") pod \"node-ca-dsp96\" (UID: \"73682abc-fb01-4ac2-a073-fcffad51a512\") " pod="openshift-image-registry/node-ca-dsp96" Nov 02 22:47:34.129297 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.127871 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-modprobe-d\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.129297 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.127895 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d05df5bf-8c7c-4c79-afa6-96fc7d3689e4-tmp-dir\") pod \"node-resolver-l5tth\" (UID: \"d05df5bf-8c7c-4c79-afa6-96fc7d3689e4\") " pod="openshift-dns/node-resolver-l5tth" Nov 02 22:47:34.129297 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.127924 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-cnibin\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.129297 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.127955 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-multus-socket-dir-parent\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.129297 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.127983 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-run-k8s-cni-cncf-io\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.129297 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.128010 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg6gf\" (UniqueName: \"kubernetes.io/projected/32856eec-17a2-4d3a-8ca1-53755398f81c-kube-api-access-pg6gf\") pod \"iptables-alerter-jvpc5\" (UID: \"32856eec-17a2-4d3a-8ca1-53755398f81c\") " pod="openshift-network-operator/iptables-alerter-jvpc5" Nov 02 22:47:34.129297 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.128032 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-run-netns\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.129297 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.128059 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-run-systemd\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.129297 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.128086 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-var-lib-kubelet\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.129297 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.128113 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-cni-bin\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.129297 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.128140 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret\") pod \"global-pull-secret-syncer-9drjv\" (UID: \"e608a3bc-a7f2-45a1-9603-d85bbbe20614\") " pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:34.130052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.128162 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-tuned\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.130052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.128191 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-etc-selinux\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:47:34.130052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.128221 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1062888a-e10c-4a34-b84a-f85ff9eb8063-konnectivity-ca\") pod \"konnectivity-agent-ftp5v\" (UID: \"1062888a-e10c-4a34-b84a-f85ff9eb8063\") " pod="kube-system/konnectivity-agent-ftp5v" Nov 02 22:47:34.130052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.128249 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-etc-kubernetes\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.130052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.128282 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-kubelet\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.130052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.128325 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-run-ovn\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.130052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.128347 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-log-socket\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.130052 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.128375 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.141781 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.141750 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hnmtx\"" Nov 02 22:47:34.141910 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.141819 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 22:47:34.159564 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.159537 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-77tzt\"" Nov 02 22:47:34.159564 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.159545 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9jqgm\"" Nov 02 22:47:34.162711 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.162592 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mb4zk\"" Nov 02 22:47:34.162711 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.162650 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 22:47:34.162968 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.162955 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 22:47:34.163001 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.162984 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 22:47:34.163145 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.163061 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 22:47:34.163145 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.163096 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:47:34.163145 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.163067 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 22:47:34.163145 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.163121 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 22:47:34.163145 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.163128 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 22:47:34.163145 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.163127 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 22:47:34.164161 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.164150 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 22:47:34.164265 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.164253 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 22:47:34.164299 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.164292 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 22:47:34.168368 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.168352 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 22:47:34.210780 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.210755 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 22:47:34.210919 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.210836 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nmk2g" Nov 02 22:47:34.228829 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.228799 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-modprobe-d\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.228829 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.228831 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d05df5bf-8c7c-4c79-afa6-96fc7d3689e4-tmp-dir\") pod \"node-resolver-l5tth\" (UID: \"d05df5bf-8c7c-4c79-afa6-96fc7d3689e4\") " pod="openshift-dns/node-resolver-l5tth" Nov 02 22:47:34.229039 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.228848 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-cnibin\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.229039 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.228865 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-multus-socket-dir-parent\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.229039 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.228880 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-run-k8s-cni-cncf-io\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.229039 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.228895 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg6gf\" (UniqueName: \"kubernetes.io/projected/32856eec-17a2-4d3a-8ca1-53755398f81c-kube-api-access-pg6gf\") pod \"iptables-alerter-jvpc5\" (UID: \"32856eec-17a2-4d3a-8ca1-53755398f81c\") " pod="openshift-network-operator/iptables-alerter-jvpc5" Nov 02 22:47:34.229039 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.228924 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-run-netns\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.229039 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.228945 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-run-systemd\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.229039 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.228991 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-run-systemd\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.229039 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.228995 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-run-netns\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.229039 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.228991 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-cnibin\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.229039 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229014 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-modprobe-d\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.229039 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229027 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-run-k8s-cni-cncf-io\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.229039 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229037 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-multus-socket-dir-parent\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.229654 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229107 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-var-lib-kubelet\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.229654 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229132 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-cni-bin\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.229654 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229148 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret\") pod \"global-pull-secret-syncer-9drjv\" (UID: \"e608a3bc-a7f2-45a1-9603-d85bbbe20614\") " pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:34.229654 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229163 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-tuned\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.229654 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229190 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-etc-selinux\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:47:34.229654 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229208 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-cni-bin\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.229654 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229217 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d05df5bf-8c7c-4c79-afa6-96fc7d3689e4-tmp-dir\") pod \"node-resolver-l5tth\" (UID: \"d05df5bf-8c7c-4c79-afa6-96fc7d3689e4\") " pod="openshift-dns/node-resolver-l5tth" Nov 02 22:47:34.229654 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229237 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1062888a-e10c-4a34-b84a-f85ff9eb8063-konnectivity-ca\") pod \"konnectivity-agent-ftp5v\" (UID: \"1062888a-e10c-4a34-b84a-f85ff9eb8063\") " pod="kube-system/konnectivity-agent-ftp5v" Nov 02 22:47:34.229654 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229252 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-var-lib-kubelet\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.229654 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229261 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-etc-kubernetes\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.229654 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:34.229257 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:34.229654 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229285 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-kubelet\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.229654 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229307 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-run-ovn\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.229654 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229345 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-etc-selinux\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:47:34.229654 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229341 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-etc-kubernetes\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.229654 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:34.229348 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret podName:e608a3bc-a7f2-45a1-9603-d85bbbe20614 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:34.729305776 +0000 UTC m=+2.255950380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret") pod "global-pull-secret-syncer-9drjv" (UID: "e608a3bc-a7f2-45a1-9603-d85bbbe20614") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:34.229654 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229346 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-run-ovn\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.230178 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229390 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-log-socket\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.230178 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229403 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-kubelet\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.230178 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229407 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.230178 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229433 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-sysconfig\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.230178 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229462 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-sysconfig\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.230178 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229460 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 22:47:34.230178 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229503 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-log-socket\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.230178 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229518 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.230178 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229517 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1062888a-e10c-4a34-b84a-f85ff9eb8063-agent-certs\") pod \"konnectivity-agent-ftp5v\" (UID: \"1062888a-e10c-4a34-b84a-f85ff9eb8063\") " pod="kube-system/konnectivity-agent-ftp5v" Nov 02 22:47:34.230178 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229548 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-var-lib-cni-bin\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.230178 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229579 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs\") pod \"network-metrics-daemon-fqwl4\" (UID: \"5ce08d97-9151-4a82-92b3-a01a4057f046\") " pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:47:34.230178 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229606 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxxkn\" (UniqueName: \"kubernetes.io/projected/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4-kube-api-access-kxxkn\") pod \"network-check-target-569mb\" (UID: \"f8acfea0-a4d3-4673-9fc6-e48a94dfbba4\") " pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:47:34.230178 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229608 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-var-lib-cni-bin\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.230178 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:34.229665 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:34.230178 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:34.229713 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs podName:5ce08d97-9151-4a82-92b3-a01a4057f046 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:34.72970222 +0000 UTC m=+2.256346824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs") pod "network-metrics-daemon-fqwl4" (UID: "5ce08d97-9151-4a82-92b3-a01a4057f046") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:34.230178 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229632 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0e82120d-ed55-489b-a3d5-a553ac1b1678-env-overrides\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.230178 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229751 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-systemd\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.231012 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229768 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-var-lib-cni-multus\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.231012 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229784 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-multus-conf-dir\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.231012 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229803 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-var-lib-cni-multus\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.231012 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229809 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-run-multus-certs\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.231012 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229831 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32856eec-17a2-4d3a-8ca1-53755398f81c-host-slash\") pod \"iptables-alerter-jvpc5\" (UID: \"32856eec-17a2-4d3a-8ca1-53755398f81c\") " pod="openshift-network-operator/iptables-alerter-jvpc5" Nov 02 22:47:34.231012 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229833 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-run-multus-certs\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.231012 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229847 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-var-lib-kubelet\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.231012 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229858 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-multus-conf-dir\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.231012 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229861 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-var-lib-openvswitch\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.231012 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229783 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-systemd\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.231012 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-sysctl-d\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.231012 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229894 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:47:34.231012 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229911 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lppg\" (UniqueName: \"kubernetes.io/projected/fabb0266-226c-4935-a9fc-5ed6199e2f9b-kube-api-access-9lppg\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:47:34.231012 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229941 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-var-lib-kubelet\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.231012 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229946 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a5e437f-4589-4f76-b777-026186f06c73-cni-binary-copy\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.231012 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229967 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-var-lib-openvswitch\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.231012 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229981 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-etc-openvswitch\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.231761 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229998 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:47:34.231761 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.229948 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-sysctl-d\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.231761 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230003 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73682abc-fb01-4ac2-a073-fcffad51a512-host\") pod \"node-ca-dsp96\" (UID: \"73682abc-fb01-4ac2-a073-fcffad51a512\") " pod="openshift-image-registry/node-ca-dsp96" Nov 02 22:47:34.231761 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230022 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-host\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.231761 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230028 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73682abc-fb01-4ac2-a073-fcffad51a512-host\") pod \"node-ca-dsp96\" (UID: \"73682abc-fb01-4ac2-a073-fcffad51a512\") " pod="openshift-image-registry/node-ca-dsp96" Nov 02 22:47:34.231761 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230038 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm5cb\" (UniqueName: \"kubernetes.io/projected/d05df5bf-8c7c-4c79-afa6-96fc7d3689e4-kube-api-access-cm5cb\") pod \"node-resolver-l5tth\" (UID: \"d05df5bf-8c7c-4c79-afa6-96fc7d3689e4\") " pod="openshift-dns/node-resolver-l5tth" Nov 02 22:47:34.231761 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230055 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcgff\" (UniqueName: \"kubernetes.io/projected/ca56f427-3734-48ec-b7ce-100daeb7b3bb-kube-api-access-pcgff\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:47:34.231761 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230068 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-etc-openvswitch\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.231761 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230073 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca56f427-3734-48ec-b7ce-100daeb7b3bb-cnibin\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:47:34.231761 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230096 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-run-ovn-kubernetes\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.231761 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230107 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-host\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.231761 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230117 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-sys\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.231761 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230138 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-sys-fs\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:47:34.231761 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230154 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32856eec-17a2-4d3a-8ca1-53755398f81c-host-slash\") pod \"iptables-alerter-jvpc5\" (UID: \"32856eec-17a2-4d3a-8ca1-53755398f81c\") " pod="openshift-network-operator/iptables-alerter-jvpc5" Nov 02 22:47:34.231761 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230161 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-system-cni-dir\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.231761 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230214 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-system-cni-dir\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.231761 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230260 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1062888a-e10c-4a34-b84a-f85ff9eb8063-konnectivity-ca\") pod \"konnectivity-agent-ftp5v\" (UID: \"1062888a-e10c-4a34-b84a-f85ff9eb8063\") " pod="kube-system/konnectivity-agent-ftp5v" Nov 02 22:47:34.231761 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a5e437f-4589-4f76-b777-026186f06c73-multus-daemon-config\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.232627 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230348 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e608a3bc-a7f2-45a1-9603-d85bbbe20614-kubelet-config\") pod \"global-pull-secret-syncer-9drjv\" (UID: \"e608a3bc-a7f2-45a1-9603-d85bbbe20614\") " pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:34.232627 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230367 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-run-ovn-kubernetes\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.232627 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230373 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-kubernetes\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.232627 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230430 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-kubernetes\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.232627 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230451 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-sys\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.232627 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230531 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-sys-fs\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:47:34.232627 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230578 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e608a3bc-a7f2-45a1-9603-d85bbbe20614-kubelet-config\") pod \"global-pull-secret-syncer-9drjv\" (UID: \"e608a3bc-a7f2-45a1-9603-d85bbbe20614\") " pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:34.232627 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230582 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca56f427-3734-48ec-b7ce-100daeb7b3bb-cnibin\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:47:34.232627 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230602 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0e82120d-ed55-489b-a3d5-a553ac1b1678-env-overrides\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.232627 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230605 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca56f427-3734-48ec-b7ce-100daeb7b3bb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:47:34.232627 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230608 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a5e437f-4589-4f76-b777-026186f06c73-cni-binary-copy\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.232627 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230654 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-socket-dir\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:47:34.232627 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230674 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpfqn\" (UniqueName: \"kubernetes.io/projected/5ce08d97-9151-4a82-92b3-a01a4057f046-kube-api-access-qpfqn\") pod \"network-metrics-daemon-fqwl4\" (UID: \"5ce08d97-9151-4a82-92b3-a01a4057f046\") " pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:47:34.232627 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0e82120d-ed55-489b-a3d5-a553ac1b1678-ovnkube-config\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.232627 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230723 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj75b\" (UniqueName: \"kubernetes.io/projected/0e82120d-ed55-489b-a3d5-a553ac1b1678-kube-api-access-sj75b\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.232627 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230745 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnvrv\" (UniqueName: \"kubernetes.io/projected/73682abc-fb01-4ac2-a073-fcffad51a512-kube-api-access-nnvrv\") pod \"node-ca-dsp96\" (UID: \"73682abc-fb01-4ac2-a073-fcffad51a512\") " pod="openshift-image-registry/node-ca-dsp96" Nov 02 22:47:34.232627 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230765 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-run\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.233406 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230790 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-lib-modules\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.233406 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230807 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca56f427-3734-48ec-b7ce-100daeb7b3bb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:47:34.233406 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230815 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32856eec-17a2-4d3a-8ca1-53755398f81c-iptables-alerter-script\") pod \"iptables-alerter-jvpc5\" (UID: \"32856eec-17a2-4d3a-8ca1-53755398f81c\") " pod="openshift-network-operator/iptables-alerter-jvpc5" Nov 02 22:47:34.233406 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230846 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-slash\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.233406 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230870 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-run-openvswitch\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.233406 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230898 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-node-log\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.233406 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230905 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a5e437f-4589-4f76-b777-026186f06c73-multus-daemon-config\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.233406 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230906 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-socket-dir\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:47:34.233406 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230920 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-cni-netd\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.233406 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230946 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e82120d-ed55-489b-a3d5-a553ac1b1678-ovn-node-metrics-cert\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.233406 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230970 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee507e21-156a-48a5-9c73-5f9e98bdd44a-tmp\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.233406 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230974 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-slash\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.233406 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.230992 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca56f427-3734-48ec-b7ce-100daeb7b3bb-cni-binary-copy\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:47:34.233406 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.231017 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca56f427-3734-48ec-b7ce-100daeb7b3bb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:47:34.233406 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.231021 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-cni-netd\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.233406 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.231042 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-systemd-units\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.233406 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.231056 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-run-openvswitch\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.233962 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.231067 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0e82120d-ed55-489b-a3d5-a553ac1b1678-ovnkube-script-lib\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.233962 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.231095 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-sysctl-conf\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.233962 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.231099 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-node-log\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.233962 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.231119 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j2w5\" (UniqueName: \"kubernetes.io/projected/ee507e21-156a-48a5-9c73-5f9e98bdd44a-kube-api-access-6j2w5\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.233962 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.231147 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca56f427-3734-48ec-b7ce-100daeb7b3bb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:47:34.233962 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.231196 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-registration-dir\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:47:34.233962 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.231237 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-run-netns\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.233962 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.231261 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-hostroot\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.233962 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.231285 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e608a3bc-a7f2-45a1-9603-d85bbbe20614-dbus\") pod \"global-pull-secret-syncer-9drjv\" (UID: \"e608a3bc-a7f2-45a1-9603-d85bbbe20614\") " pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:34.233962 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.231307 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d05df5bf-8c7c-4c79-afa6-96fc7d3689e4-hosts-file\") pod \"node-resolver-l5tth\" (UID: \"d05df5bf-8c7c-4c79-afa6-96fc7d3689e4\") " pod="openshift-dns/node-resolver-l5tth" Nov 02 22:47:34.233962 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.231331 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca56f427-3734-48ec-b7ce-100daeb7b3bb-system-cni-dir\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:47:34.233962 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.231357 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-device-dir\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:47:34.233962 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.231382 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-multus-cni-dir\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.233962 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.231412 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66b2k\" (UniqueName: \"kubernetes.io/projected/0a5e437f-4589-4f76-b777-026186f06c73-kube-api-access-66b2k\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.233962 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.231455 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/73682abc-fb01-4ac2-a073-fcffad51a512-serviceca\") pod \"node-ca-dsp96\" (UID: \"73682abc-fb01-4ac2-a073-fcffad51a512\") " pod="openshift-image-registry/node-ca-dsp96" Nov 02 22:47:34.233962 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.231535 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32856eec-17a2-4d3a-8ca1-53755398f81c-iptables-alerter-script\") pod \"iptables-alerter-jvpc5\" (UID: \"32856eec-17a2-4d3a-8ca1-53755398f81c\") " pod="openshift-network-operator/iptables-alerter-jvpc5" Nov 02 22:47:34.233962 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.231581 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-systemd-units\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.234439 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.232081 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca56f427-3734-48ec-b7ce-100daeb7b3bb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:47:34.234439 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.232127 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0e82120d-ed55-489b-a3d5-a553ac1b1678-ovnkube-config\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.234439 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.232361 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/73682abc-fb01-4ac2-a073-fcffad51a512-serviceca\") pod \"node-ca-dsp96\" (UID: \"73682abc-fb01-4ac2-a073-fcffad51a512\") " pod="openshift-image-registry/node-ca-dsp96" Nov 02 22:47:34.234439 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.232577 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca56f427-3734-48ec-b7ce-100daeb7b3bb-cni-binary-copy\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:47:34.234439 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.232650 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-run\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.234439 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.232703 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-hostroot\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.234439 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.232769 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-sysctl-conf\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.234439 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.232936 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0e82120d-ed55-489b-a3d5-a553ac1b1678-ovnkube-script-lib\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.234439 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.232999 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca56f427-3734-48ec-b7ce-100daeb7b3bb-system-cni-dir\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:47:34.234439 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.233120 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e608a3bc-a7f2-45a1-9603-d85bbbe20614-dbus\") pod \"global-pull-secret-syncer-9drjv\" (UID: \"e608a3bc-a7f2-45a1-9603-d85bbbe20614\") " pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:34.234439 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.233124 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca56f427-3734-48ec-b7ce-100daeb7b3bb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:47:34.234439 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.233173 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-device-dir\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:47:34.234439 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.233189 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d05df5bf-8c7c-4c79-afa6-96fc7d3689e4-hosts-file\") pod \"node-resolver-l5tth\" (UID: \"d05df5bf-8c7c-4c79-afa6-96fc7d3689e4\") " pod="openshift-dns/node-resolver-l5tth" Nov 02 22:47:34.234439 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.233191 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-multus-cni-dir\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.234439 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.233234 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-run-netns\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.234439 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.233238 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-registration-dir\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:47:34.234439 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.231148 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-lib-modules\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.234439 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.233730 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-tuned\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.235088 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.233840 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1062888a-e10c-4a34-b84a-f85ff9eb8063-agent-certs\") pod \"konnectivity-agent-ftp5v\" (UID: \"1062888a-e10c-4a34-b84a-f85ff9eb8063\") " pod="kube-system/konnectivity-agent-ftp5v" Nov 02 22:47:34.235088 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.234710 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e82120d-ed55-489b-a3d5-a553ac1b1678-ovn-node-metrics-cert\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.235088 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.234925 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee507e21-156a-48a5-9c73-5f9e98bdd44a-tmp\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.287057 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:34.286963 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:47:34.287057 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:34.286995 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:47:34.287057 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:34.287011 2578 projected.go:196] Error preparing data for projected volume kube-api-access-kxxkn for pod openshift-network-diagnostics/network-check-target-569mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:34.287265 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:34.287084 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4-kube-api-access-kxxkn podName:f8acfea0-a4d3-4673-9fc6-e48a94dfbba4 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:34.787064629 +0000 UTC m=+2.313709248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kxxkn" (UniqueName: "kubernetes.io/projected/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4-kube-api-access-kxxkn") pod "network-check-target-569mb" (UID: "f8acfea0-a4d3-4673-9fc6-e48a94dfbba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:34.290178 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.290150 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lppg\" (UniqueName: \"kubernetes.io/projected/fabb0266-226c-4935-a9fc-5ed6199e2f9b-kube-api-access-9lppg\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:47:34.290296 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.290173 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg6gf\" (UniqueName: \"kubernetes.io/projected/32856eec-17a2-4d3a-8ca1-53755398f81c-kube-api-access-pg6gf\") pod \"iptables-alerter-jvpc5\" (UID: \"32856eec-17a2-4d3a-8ca1-53755398f81c\") " pod="openshift-network-operator/iptables-alerter-jvpc5" Nov 02 22:47:34.290375 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.290354 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm5cb\" (UniqueName: \"kubernetes.io/projected/d05df5bf-8c7c-4c79-afa6-96fc7d3689e4-kube-api-access-cm5cb\") pod \"node-resolver-l5tth\" (UID: \"d05df5bf-8c7c-4c79-afa6-96fc7d3689e4\") " pod="openshift-dns/node-resolver-l5tth" Nov 02 22:47:34.290580 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.290564 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpfqn\" (UniqueName: \"kubernetes.io/projected/5ce08d97-9151-4a82-92b3-a01a4057f046-kube-api-access-qpfqn\") pod \"network-metrics-daemon-fqwl4\" (UID: \"5ce08d97-9151-4a82-92b3-a01a4057f046\") " pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:47:34.290837 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.290815 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcgff\" (UniqueName: \"kubernetes.io/projected/ca56f427-3734-48ec-b7ce-100daeb7b3bb-kube-api-access-pcgff\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:47:34.294966 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.294936 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj75b\" (UniqueName: \"kubernetes.io/projected/0e82120d-ed55-489b-a3d5-a553ac1b1678-kube-api-access-sj75b\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.295121 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.295103 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66b2k\" (UniqueName: \"kubernetes.io/projected/0a5e437f-4589-4f76-b777-026186f06c73-kube-api-access-66b2k\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.295579 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.295561 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j2w5\" (UniqueName: \"kubernetes.io/projected/ee507e21-156a-48a5-9c73-5f9e98bdd44a-kube-api-access-6j2w5\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.295865 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.295851 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnvrv\" (UniqueName: \"kubernetes.io/projected/73682abc-fb01-4ac2-a073-fcffad51a512-kube-api-access-nnvrv\") pod \"node-ca-dsp96\" (UID: \"73682abc-fb01-4ac2-a073-fcffad51a512\") " pod="openshift-image-registry/node-ca-dsp96" Nov 02 22:47:34.403680 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.403639 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:47:34.404690 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.404663 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dsp96" Nov 02 22:47:34.434683 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.434649 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ftp5v" Nov 02 22:47:34.453668 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.453642 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:47:34.462427 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.462388 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l5tth" Nov 02 22:47:34.492351 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.492316 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:47:34.496026 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.496006 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hg78p" Nov 02 22:47:34.512160 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.512140 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jvpc5" Nov 02 22:47:34.522322 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:34.522277 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5e437f_4589_4f76_b777_026186f06c73.slice/crio-754fedf886b43368b928ff4a1d861296eb55a85d000114da6a782f4f41fbdf89 WatchSource:0}: Error finding container 754fedf886b43368b928ff4a1d861296eb55a85d000114da6a782f4f41fbdf89: Status 404 returned error can't find the container with id 754fedf886b43368b928ff4a1d861296eb55a85d000114da6a782f4f41fbdf89 Nov 02 22:47:34.522600 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:34.522569 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca56f427_3734_48ec_b7ce_100daeb7b3bb.slice/crio-5d00480a202d2e0f59c4e97f43655a9ebc9742638459c8d5eac352dc1c64aee5 WatchSource:0}: Error finding container 5d00480a202d2e0f59c4e97f43655a9ebc9742638459c8d5eac352dc1c64aee5: Status 404 returned error can't find the container with id 5d00480a202d2e0f59c4e97f43655a9ebc9742638459c8d5eac352dc1c64aee5 Nov 02 22:47:34.523230 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:34.523207 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6e8e95c46679d654a23b98c776960b.slice/crio-fc5798118300a384cf4aba07c07566f07b52867f51baaf40da43e1569a39264c WatchSource:0}: Error finding container fc5798118300a384cf4aba07c07566f07b52867f51baaf40da43e1569a39264c: Status 404 returned error can't find the container with id fc5798118300a384cf4aba07c07566f07b52867f51baaf40da43e1569a39264c Nov 02 22:47:34.523633 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:34.523601 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09bea7df1640e9a4e82191575067a028.slice/crio-8fd672dc0cc08aad560279c4603970941fb1575285bcbea70018da387a77c42a WatchSource:0}: Error finding container 8fd672dc0cc08aad560279c4603970941fb1575285bcbea70018da387a77c42a: Status 404 returned error can't find the container with id 8fd672dc0cc08aad560279c4603970941fb1575285bcbea70018da387a77c42a Nov 02 22:47:34.526121 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:34.526101 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32856eec_17a2_4d3a_8ca1_53755398f81c.slice/crio-3eff20b4dd4839ad801ecc8d1fd85f714de2dc60127bd547b37c0912bfafb0ef WatchSource:0}: Error finding container 3eff20b4dd4839ad801ecc8d1fd85f714de2dc60127bd547b37c0912bfafb0ef: Status 404 returned error can't find the container with id 3eff20b4dd4839ad801ecc8d1fd85f714de2dc60127bd547b37c0912bfafb0ef Nov 02 22:47:34.528239 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.528221 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:47:34.534077 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.534055 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:34.543877 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:34.543853 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e82120d_ed55_489b_a3d5_a553ac1b1678.slice/crio-1891a2fee619b7749bd605bce66982e33fdf968cd5798254a682a85b5afe31a2 WatchSource:0}: Error finding container 1891a2fee619b7749bd605bce66982e33fdf968cd5798254a682a85b5afe31a2: Status 404 returned error can't find the container with id 1891a2fee619b7749bd605bce66982e33fdf968cd5798254a682a85b5afe31a2 Nov 02 22:47:34.735193 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.735154 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret\") pod \"global-pull-secret-syncer-9drjv\" (UID: \"e608a3bc-a7f2-45a1-9603-d85bbbe20614\") " pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:34.735193 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.735202 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs\") pod \"network-metrics-daemon-fqwl4\" (UID: \"5ce08d97-9151-4a82-92b3-a01a4057f046\") " pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:47:34.735407 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:34.735333 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:34.735407 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:34.735336 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:34.735407 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:34.735391 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs podName:5ce08d97-9151-4a82-92b3-a01a4057f046 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:35.735376459 +0000 UTC m=+3.262021063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs") pod "network-metrics-daemon-fqwl4" (UID: "5ce08d97-9151-4a82-92b3-a01a4057f046") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:34.735407 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:34.735405 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret podName:e608a3bc-a7f2-45a1-9603-d85bbbe20614 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:35.73539831 +0000 UTC m=+3.262042914 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret") pod "global-pull-secret-syncer-9drjv" (UID: "e608a3bc-a7f2-45a1-9603-d85bbbe20614") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:34.836213 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.836122 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxxkn\" (UniqueName: \"kubernetes.io/projected/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4-kube-api-access-kxxkn\") pod \"network-check-target-569mb\" (UID: \"f8acfea0-a4d3-4673-9fc6-e48a94dfbba4\") " pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:47:34.836895 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:34.836309 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:47:34.836895 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:34.836338 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:47:34.836895 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:34.836350 2578 projected.go:196] Error preparing data for projected volume kube-api-access-kxxkn for pod openshift-network-diagnostics/network-check-target-569mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:34.836895 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:34.836409 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4-kube-api-access-kxxkn podName:f8acfea0-a4d3-4673-9fc6-e48a94dfbba4 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:35.836391231 +0000 UTC m=+3.363035850 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kxxkn" (UniqueName: "kubernetes.io/projected/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4-kube-api-access-kxxkn") pod "network-check-target-569mb" (UID: "f8acfea0-a4d3-4673-9fc6-e48a94dfbba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:34.973463 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.973429 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:47:34.975846 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.975799 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" event={"ID":"0e82120d-ed55-489b-a3d5-a553ac1b1678","Type":"ContainerStarted","Data":"1891a2fee619b7749bd605bce66982e33fdf968cd5798254a682a85b5afe31a2"} Nov 02 22:47:34.976923 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.976892 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jvpc5" event={"ID":"32856eec-17a2-4d3a-8ca1-53755398f81c","Type":"ContainerStarted","Data":"3eff20b4dd4839ad801ecc8d1fd85f714de2dc60127bd547b37c0912bfafb0ef"} Nov 02 22:47:34.977957 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.977919 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal" event={"ID":"09bea7df1640e9a4e82191575067a028","Type":"ContainerStarted","Data":"8fd672dc0cc08aad560279c4603970941fb1575285bcbea70018da387a77c42a"} Nov 02 22:47:34.979083 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.979051 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpqcq" event={"ID":"ca56f427-3734-48ec-b7ce-100daeb7b3bb","Type":"ContainerStarted","Data":"5d00480a202d2e0f59c4e97f43655a9ebc9742638459c8d5eac352dc1c64aee5"} Nov 02 22:47:34.980204 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.980184 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hg78p" event={"ID":"0a5e437f-4589-4f76-b777-026186f06c73","Type":"ContainerStarted","Data":"754fedf886b43368b928ff4a1d861296eb55a85d000114da6a782f4f41fbdf89"} Nov 02 22:47:34.981212 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:34.981187 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" event={"ID":"7c6e8e95c46679d654a23b98c776960b","Type":"ContainerStarted","Data":"fc5798118300a384cf4aba07c07566f07b52867f51baaf40da43e1569a39264c"} Nov 02 22:47:35.160863 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:35.160770 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:47:35.214573 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:35.214458 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 22:42:34 +0000 UTC" deadline="2028-05-28 09:32:50.222377929 +0000 UTC" Nov 02 22:47:35.214573 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:35.214549 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13738h45m15.007833282s" Nov 02 22:47:35.250899 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:35.250870 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73682abc_fb01_4ac2_a073_fcffad51a512.slice/crio-e93ae72ab7af8d8f988c835df9607896d50473144bc74bfb5a4d8038e771102d WatchSource:0}: Error finding container e93ae72ab7af8d8f988c835df9607896d50473144bc74bfb5a4d8038e771102d: Status 404 returned error can't find the container with id e93ae72ab7af8d8f988c835df9607896d50473144bc74bfb5a4d8038e771102d Nov 02 22:47:35.325672 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:35.325636 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee507e21_156a_48a5_9c73_5f9e98bdd44a.slice/crio-0cd570346bfa2647390cf60c26ce8e507f7ff60662c7f4f003c76c13352fdf54 WatchSource:0}: Error finding container 0cd570346bfa2647390cf60c26ce8e507f7ff60662c7f4f003c76c13352fdf54: Status 404 returned error can't find the container with id 0cd570346bfa2647390cf60c26ce8e507f7ff60662c7f4f003c76c13352fdf54 Nov 02 22:47:35.376941 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:35.376894 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfabb0266_226c_4935_a9fc_5ed6199e2f9b.slice/crio-9a6201c52a6605c5c878d4d4889bec6411be08be8372537ce6b6e5fc1c29f333 WatchSource:0}: Error finding container 9a6201c52a6605c5c878d4d4889bec6411be08be8372537ce6b6e5fc1c29f333: Status 404 returned error can't find the container with id 9a6201c52a6605c5c878d4d4889bec6411be08be8372537ce6b6e5fc1c29f333 Nov 02 22:47:35.377452 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:35.377393 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05df5bf_8c7c_4c79_afa6_96fc7d3689e4.slice/crio-90ca4505d28c37a4e751200f1e0076d676713d8b2963645e83c2e23b434563e6 WatchSource:0}: Error finding container 90ca4505d28c37a4e751200f1e0076d676713d8b2963645e83c2e23b434563e6: Status 404 returned error can't find the container with id 90ca4505d28c37a4e751200f1e0076d676713d8b2963645e83c2e23b434563e6 Nov 02 22:47:35.396714 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:35.396673 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:47:35.403894 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:47:35.403860 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1062888a_e10c_4a34_b84a_f85ff9eb8063.slice/crio-4468bd03a890b8fae2c6d143db0a84a5e10601fc853fd2ae6e9c4c61b5481300 WatchSource:0}: Error finding container 4468bd03a890b8fae2c6d143db0a84a5e10601fc853fd2ae6e9c4c61b5481300: Status 404 returned error can't find the container with id 4468bd03a890b8fae2c6d143db0a84a5e10601fc853fd2ae6e9c4c61b5481300 Nov 02 22:47:35.744543 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:35.744483 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret\") pod \"global-pull-secret-syncer-9drjv\" (UID: \"e608a3bc-a7f2-45a1-9603-d85bbbe20614\") " pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:35.744543 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:35.744560 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs\") pod \"network-metrics-daemon-fqwl4\" (UID: \"5ce08d97-9151-4a82-92b3-a01a4057f046\") " pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:47:35.744837 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:35.744689 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:35.744837 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:35.744689 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:35.744837 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:35.744796 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret podName:e608a3bc-a7f2-45a1-9603-d85bbbe20614 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:37.744738734 +0000 UTC m=+5.271383339 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret") pod "global-pull-secret-syncer-9drjv" (UID: "e608a3bc-a7f2-45a1-9603-d85bbbe20614") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:35.744837 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:35.744826 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs podName:5ce08d97-9151-4a82-92b3-a01a4057f046 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:37.744812613 +0000 UTC m=+5.271457233 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs") pod "network-metrics-daemon-fqwl4" (UID: "5ce08d97-9151-4a82-92b3-a01a4057f046") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:35.845946 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:35.845902 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxxkn\" (UniqueName: \"kubernetes.io/projected/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4-kube-api-access-kxxkn\") pod \"network-check-target-569mb\" (UID: \"f8acfea0-a4d3-4673-9fc6-e48a94dfbba4\") " pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:47:35.846730 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:35.846102 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:47:35.846730 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:35.846124 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:47:35.846730 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:35.846138 2578 projected.go:196] Error preparing data for projected volume kube-api-access-kxxkn for pod openshift-network-diagnostics/network-check-target-569mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:35.846730 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:35.846243 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4-kube-api-access-kxxkn podName:f8acfea0-a4d3-4673-9fc6-e48a94dfbba4 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:37.846221343 +0000 UTC m=+5.372865954 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kxxkn" (UniqueName: "kubernetes.io/projected/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4-kube-api-access-kxxkn") pod "network-check-target-569mb" (UID: "f8acfea0-a4d3-4673-9fc6-e48a94dfbba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:35.972322 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:35.971814 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:47:35.972322 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:35.971936 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-569mb" podUID="f8acfea0-a4d3-4673-9fc6-e48a94dfbba4" Nov 02 22:47:35.972322 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:35.972037 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:47:35.972322 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:35.972116 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqwl4" podUID="5ce08d97-9151-4a82-92b3-a01a4057f046" Nov 02 22:47:35.972322 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:35.972171 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:35.972322 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:35.972236 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9drjv" podUID="e608a3bc-a7f2-45a1-9603-d85bbbe20614" Nov 02 22:47:35.985831 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:35.985789 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ftp5v" event={"ID":"1062888a-e10c-4a34-b84a-f85ff9eb8063","Type":"ContainerStarted","Data":"4468bd03a890b8fae2c6d143db0a84a5e10601fc853fd2ae6e9c4c61b5481300"} Nov 02 22:47:35.989733 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:35.989691 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l5tth" event={"ID":"d05df5bf-8c7c-4c79-afa6-96fc7d3689e4","Type":"ContainerStarted","Data":"90ca4505d28c37a4e751200f1e0076d676713d8b2963645e83c2e23b434563e6"} Nov 02 22:47:35.993105 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:35.993007 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" event={"ID":"fabb0266-226c-4935-a9fc-5ed6199e2f9b","Type":"ContainerStarted","Data":"9a6201c52a6605c5c878d4d4889bec6411be08be8372537ce6b6e5fc1c29f333"} Nov 02 22:47:35.999287 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:35.999206 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9pf69" event={"ID":"ee507e21-156a-48a5-9c73-5f9e98bdd44a","Type":"ContainerStarted","Data":"0cd570346bfa2647390cf60c26ce8e507f7ff60662c7f4f003c76c13352fdf54"} Nov 02 22:47:36.003077 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:36.003046 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dsp96" event={"ID":"73682abc-fb01-4ac2-a073-fcffad51a512","Type":"ContainerStarted","Data":"e93ae72ab7af8d8f988c835df9607896d50473144bc74bfb5a4d8038e771102d"} Nov 02 22:47:36.215704 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:36.215645 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 22:42:34 +0000 UTC" deadline="2028-04-09 05:19:53.986100862 +0000 UTC" Nov 02 22:47:36.215704 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:36.215692 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12558h32m17.770413314s" Nov 02 22:47:37.761362 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:37.761321 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs\") pod \"network-metrics-daemon-fqwl4\" (UID: \"5ce08d97-9151-4a82-92b3-a01a4057f046\") " pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:47:37.761901 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:37.761428 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret\") pod \"global-pull-secret-syncer-9drjv\" (UID: \"e608a3bc-a7f2-45a1-9603-d85bbbe20614\") " pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:37.761901 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:37.761595 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:37.761901 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:37.761665 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret podName:e608a3bc-a7f2-45a1-9603-d85bbbe20614 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:41.761645679 +0000 UTC m=+9.288290303 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret") pod "global-pull-secret-syncer-9drjv" (UID: "e608a3bc-a7f2-45a1-9603-d85bbbe20614") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:37.761901 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:37.761745 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:37.761901 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:37.761785 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs podName:5ce08d97-9151-4a82-92b3-a01a4057f046 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:41.761772079 +0000 UTC m=+9.288416703 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs") pod "network-metrics-daemon-fqwl4" (UID: "5ce08d97-9151-4a82-92b3-a01a4057f046") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:37.861983 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:37.861896 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxxkn\" (UniqueName: \"kubernetes.io/projected/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4-kube-api-access-kxxkn\") pod \"network-check-target-569mb\" (UID: \"f8acfea0-a4d3-4673-9fc6-e48a94dfbba4\") " pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:47:37.862188 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:37.862084 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:47:37.862188 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:37.862112 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:47:37.862188 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:37.862126 2578 projected.go:196] Error preparing data for projected volume kube-api-access-kxxkn for pod openshift-network-diagnostics/network-check-target-569mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:37.862407 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:37.862190 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4-kube-api-access-kxxkn podName:f8acfea0-a4d3-4673-9fc6-e48a94dfbba4 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:41.862171549 +0000 UTC m=+9.388816172 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kxxkn" (UniqueName: "kubernetes.io/projected/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4-kube-api-access-kxxkn") pod "network-check-target-569mb" (UID: "f8acfea0-a4d3-4673-9fc6-e48a94dfbba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:37.971536 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:37.971477 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:47:37.971761 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:37.971638 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-569mb" podUID="f8acfea0-a4d3-4673-9fc6-e48a94dfbba4" Nov 02 22:47:37.971761 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:37.971735 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:37.971948 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:37.971810 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9drjv" podUID="e608a3bc-a7f2-45a1-9603-d85bbbe20614" Nov 02 22:47:37.971948 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:37.971866 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:47:37.971948 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:37.971961 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqwl4" podUID="5ce08d97-9151-4a82-92b3-a01a4057f046" Nov 02 22:47:39.039517 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:39.039260 2578 generic.go:356] "Generic (PLEG): container finished" podID="ca56f427-3734-48ec-b7ce-100daeb7b3bb" containerID="fcb10083c0bba9d367fdb7df95b93b7af2b34efb9fa61d2ecad9527722b49c2a" exitCode=0 Nov 02 22:47:39.040007 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:39.039478 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpqcq" event={"ID":"ca56f427-3734-48ec-b7ce-100daeb7b3bb","Type":"ContainerDied","Data":"fcb10083c0bba9d367fdb7df95b93b7af2b34efb9fa61d2ecad9527722b49c2a"} Nov 02 22:47:39.971331 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:39.971241 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:47:39.971577 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:39.971398 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqwl4" podUID="5ce08d97-9151-4a82-92b3-a01a4057f046" Nov 02 22:47:39.971577 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:39.971469 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:47:39.971577 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:39.971570 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-569mb" podUID="f8acfea0-a4d3-4673-9fc6-e48a94dfbba4" Nov 02 22:47:39.971770 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:39.971622 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:39.971770 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:39.971714 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9drjv" podUID="e608a3bc-a7f2-45a1-9603-d85bbbe20614" Nov 02 22:47:40.052226 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:40.051471 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal" event={"ID":"09bea7df1640e9a4e82191575067a028","Type":"ContainerStarted","Data":"8c9e0bb9a88617f90d531a1ff935b24f82d3062f8a2dfe088706577583f6eb0b"} Nov 02 22:47:40.057188 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:40.056473 2578 generic.go:356] "Generic (PLEG): container finished" podID="7c6e8e95c46679d654a23b98c776960b" containerID="6f0c8168b3e8c62fa7327bf821e4d7c471d15249703638dd6f8b60a082211254" exitCode=0 Nov 02 22:47:40.057188 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:40.056544 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" event={"ID":"7c6e8e95c46679d654a23b98c776960b","Type":"ContainerDied","Data":"6f0c8168b3e8c62fa7327bf821e4d7c471d15249703638dd6f8b60a082211254"} Nov 02 22:47:40.174485 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:40.173871 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal" podStartSLOduration=7.173849765 podStartE2EDuration="7.173849765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:40.172843938 +0000 UTC m=+7.699488567" watchObservedRunningTime="2026-11-02 22:47:40.173849765 +0000 UTC m=+7.700494395" Nov 02 22:47:41.072854 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:41.072314 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" event={"ID":"7c6e8e95c46679d654a23b98c776960b","Type":"ContainerStarted","Data":"5d43e1f41298e458edf771adba427272cbf4d9b461d3e732975317b3f66e656c"} Nov 02 22:47:41.075997 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:41.075783 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ftp5v" event={"ID":"1062888a-e10c-4a34-b84a-f85ff9eb8063","Type":"ContainerStarted","Data":"b4c588c394048418e58093394ed0a444a6ec04a50deca94236a35828adc9bea1"} Nov 02 22:47:41.322990 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:41.322871 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" podStartSLOduration=8.322848065 podStartE2EDuration="8.322848065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:41.199793905 +0000 UTC m=+8.726438538" watchObservedRunningTime="2026-11-02 22:47:41.322848065 +0000 UTC m=+8.849492694" Nov 02 22:47:41.323164 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:41.322993 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ftp5v" podStartSLOduration=3.302234167 podStartE2EDuration="8.322983545s" totalImagesPullingTime="5.020749378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:41.322074495 +0000 UTC m=+8.848719127" watchObservedRunningTime="2026-11-02 22:47:41.322983545 +0000 UTC m=+8.849628176" Nov 02 22:47:41.796802 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:41.796748 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret\") pod \"global-pull-secret-syncer-9drjv\" (UID: \"e608a3bc-a7f2-45a1-9603-d85bbbe20614\") " pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:41.797008 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:41.796814 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs\") pod \"network-metrics-daemon-fqwl4\" (UID: \"5ce08d97-9151-4a82-92b3-a01a4057f046\") " pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:47:41.797084 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:41.797011 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:41.797140 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:41.797084 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs podName:5ce08d97-9151-4a82-92b3-a01a4057f046 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:49.797064255 +0000 UTC m=+17.323708885 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs") pod "network-metrics-daemon-fqwl4" (UID: "5ce08d97-9151-4a82-92b3-a01a4057f046") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:41.797212 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:41.797172 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:41.797264 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:41.797214 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret podName:e608a3bc-a7f2-45a1-9603-d85bbbe20614 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:49.79720042 +0000 UTC m=+17.323845044 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret") pod "global-pull-secret-syncer-9drjv" (UID: "e608a3bc-a7f2-45a1-9603-d85bbbe20614") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:41.897548 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:41.897436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxxkn\" (UniqueName: \"kubernetes.io/projected/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4-kube-api-access-kxxkn\") pod \"network-check-target-569mb\" (UID: \"f8acfea0-a4d3-4673-9fc6-e48a94dfbba4\") " pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:47:41.897722 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:41.897652 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:47:41.897722 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:41.897684 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:47:41.897722 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:41.897698 2578 projected.go:196] Error preparing data for projected volume kube-api-access-kxxkn for pod openshift-network-diagnostics/network-check-target-569mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:41.897900 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:41.897762 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4-kube-api-access-kxxkn podName:f8acfea0-a4d3-4673-9fc6-e48a94dfbba4 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:49.897742172 +0000 UTC m=+17.424386779 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kxxkn" (UniqueName: "kubernetes.io/projected/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4-kube-api-access-kxxkn") pod "network-check-target-569mb" (UID: "f8acfea0-a4d3-4673-9fc6-e48a94dfbba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:41.971903 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:41.971863 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:41.972114 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:41.972020 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9drjv" podUID="e608a3bc-a7f2-45a1-9603-d85bbbe20614" Nov 02 22:47:41.972198 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:41.972171 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:47:41.972407 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:41.972384 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqwl4" podUID="5ce08d97-9151-4a82-92b3-a01a4057f046" Nov 02 22:47:41.972585 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:41.972546 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:47:41.972668 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:41.972649 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-569mb" podUID="f8acfea0-a4d3-4673-9fc6-e48a94dfbba4" Nov 02 22:47:42.084472 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:42.084432 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dsp96" event={"ID":"73682abc-fb01-4ac2-a073-fcffad51a512","Type":"ContainerStarted","Data":"d78e1a30415396722fa2f9aefa441aea41752fae2b7e03b782a168f7915cc401"} Nov 02 22:47:43.089999 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:43.089914 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" event={"ID":"fabb0266-226c-4935-a9fc-5ed6199e2f9b","Type":"ContainerStarted","Data":"cc813ee792296469e2f0fc5e579c2c00ca7ad96efa359148b2faa3a05f282f05"} Nov 02 22:47:43.525390 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:43.525312 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ftp5v" Nov 02 22:47:43.526091 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:43.526060 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ftp5v" Nov 02 22:47:43.640451 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:43.640387 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dsp96" podStartSLOduration=4.417879838 podStartE2EDuration="10.640371941s" totalImagesPullingTime="6.222492103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:42.206882245 +0000 UTC m=+9.733526875" watchObservedRunningTime="2026-11-02 22:47:43.640371941 +0000 UTC m=+11.167016566" Nov 02 22:47:43.972756 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:43.971881 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:47:43.972756 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:43.972013 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-569mb" podUID="f8acfea0-a4d3-4673-9fc6-e48a94dfbba4" Nov 02 22:47:43.972756 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:43.972089 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:43.972756 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:43.972164 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9drjv" podUID="e608a3bc-a7f2-45a1-9603-d85bbbe20614" Nov 02 22:47:43.972756 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:43.972223 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:47:43.972756 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:43.972304 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqwl4" podUID="5ce08d97-9151-4a82-92b3-a01a4057f046" Nov 02 22:47:44.074823 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:44.074593 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ftp5v" Nov 02 22:47:44.075283 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:44.075086 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ftp5v" Nov 02 22:47:44.096670 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:44.096637 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hg78p" event={"ID":"0a5e437f-4589-4f76-b777-026186f06c73","Type":"ContainerStarted","Data":"0863ed9c8d6fbd8ef2c00b96a9ba8ed1b3afa7d4d3731a083be5fd679c7b1ecc"} Nov 02 22:47:44.306779 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:44.306720 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-hg78p" podStartSLOduration=2.292474908 podStartE2EDuration="11.306700873s" totalImagesPullingTime="9.014225965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:44.306279954 +0000 UTC m=+11.832924584" watchObservedRunningTime="2026-11-02 22:47:44.306700873 +0000 UTC m=+11.833345499" Nov 02 22:47:44.736026 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:44.735998 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 22:47:44.908852 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:44.908687 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T22:47:44.736019877Z","UUID":"ebb391d4-d9a2-47d6-8a75-6c3da185a16a","Handler":null,"Name":"","Endpoint":""} Nov 02 22:47:44.911191 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:44.911166 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 22:47:44.911191 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:44.911196 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 22:47:45.100639 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:45.100606 2578 generic.go:356] "Generic (PLEG): container finished" podID="ca56f427-3734-48ec-b7ce-100daeb7b3bb" containerID="426023fab7f19abe44365501205abdbc26ebc64395d885c789ea721f45725f74" exitCode=0 Nov 02 22:47:45.101133 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:45.100762 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpqcq" event={"ID":"ca56f427-3734-48ec-b7ce-100daeb7b3bb","Type":"ContainerDied","Data":"426023fab7f19abe44365501205abdbc26ebc64395d885c789ea721f45725f74"} Nov 02 22:47:45.103084 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:45.103047 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l5tth" event={"ID":"d05df5bf-8c7c-4c79-afa6-96fc7d3689e4","Type":"ContainerStarted","Data":"3fea92ac8bf1c31e57dd9d8fb0f93c4cb1f96ee41326ee5563b970ba5d5c5b25"} Nov 02 22:47:45.107118 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:45.107081 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" event={"ID":"fabb0266-226c-4935-a9fc-5ed6199e2f9b","Type":"ContainerStarted","Data":"9289d830f5a58b38299dafa5556de75e2c3deffe627309e2a519bc0c41eec0c4"} Nov 02 22:47:45.334328 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:45.334246 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-l5tth" podStartSLOduration=3.234577808 podStartE2EDuration="12.33423047s" totalImagesPullingTime="9.099652662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:45.333727334 +0000 UTC m=+12.860371962" watchObservedRunningTime="2026-11-02 22:47:45.33423047 +0000 UTC m=+12.860875094" Nov 02 22:47:45.971940 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:45.971902 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:47:45.972135 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:45.972040 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqwl4" podUID="5ce08d97-9151-4a82-92b3-a01a4057f046" Nov 02 22:47:45.972449 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:45.972429 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:47:45.972587 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:45.972547 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-569mb" podUID="f8acfea0-a4d3-4673-9fc6-e48a94dfbba4" Nov 02 22:47:45.972650 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:45.972606 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:45.972717 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:45.972692 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9drjv" podUID="e608a3bc-a7f2-45a1-9603-d85bbbe20614" Nov 02 22:47:46.113252 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:46.113214 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jvpc5" event={"ID":"32856eec-17a2-4d3a-8ca1-53755398f81c","Type":"ContainerStarted","Data":"c084cc63608b6e1281577e9e6b4d6b682be2755c651973c49b6cb47fb55e129c"} Nov 02 22:47:46.225702 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:46.225598 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jvpc5" podStartSLOduration=3.956521051 podStartE2EDuration="13.225584283s" totalImagesPullingTime="9.269063232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:46.225325197 +0000 UTC m=+13.751969822" watchObservedRunningTime="2026-11-02 22:47:46.225584283 +0000 UTC m=+13.752228907" Nov 02 22:47:47.116444 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:47.116172 2578 generic.go:356] "Generic (PLEG): container finished" podID="ca56f427-3734-48ec-b7ce-100daeb7b3bb" containerID="253e35fdd9b1b91d69fffb9897a1e25a724d5695b87e492f4d6ff3af8a79939c" exitCode=0 Nov 02 22:47:47.117038 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:47.116230 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpqcq" event={"ID":"ca56f427-3734-48ec-b7ce-100daeb7b3bb","Type":"ContainerDied","Data":"253e35fdd9b1b91d69fffb9897a1e25a724d5695b87e492f4d6ff3af8a79939c"} Nov 02 22:47:47.119160 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:47.119114 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" event={"ID":"fabb0266-226c-4935-a9fc-5ed6199e2f9b","Type":"ContainerStarted","Data":"338a84dc06d92abb113931b26c23b6e3b21e4fb9ce8449ad2e40429162a22f20"} Nov 02 22:47:47.121000 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:47.120973 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9pf69" event={"ID":"ee507e21-156a-48a5-9c73-5f9e98bdd44a","Type":"ContainerStarted","Data":"c8c5324a5e785743ac6b475c1e160160ecdf06e6f99611152ee4d14625a7d126"} Nov 02 22:47:47.364220 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:47.364164 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" podStartSLOduration=3.516207106 podStartE2EDuration="14.364149716s" totalImagesPullingTime="10.84794261s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:47.363939421 +0000 UTC m=+14.890584045" watchObservedRunningTime="2026-11-02 22:47:47.364149716 +0000 UTC m=+14.890794717" Nov 02 22:47:47.480179 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:47.480075 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9pf69" podStartSLOduration=3.543870501 podStartE2EDuration="14.480058924s" totalImagesPullingTime="10.936188423s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:47.479586026 +0000 UTC m=+15.006230652" watchObservedRunningTime="2026-11-02 22:47:47.480058924 +0000 UTC m=+15.006703549" Nov 02 22:47:47.971986 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:47.971904 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:47.971986 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:47.971982 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:47:47.972203 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:47.971982 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:47:47.972203 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:47.972105 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqwl4" podUID="5ce08d97-9151-4a82-92b3-a01a4057f046" Nov 02 22:47:47.972299 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:47.972205 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9drjv" podUID="e608a3bc-a7f2-45a1-9603-d85bbbe20614" Nov 02 22:47:47.972344 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:47.972308 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-569mb" podUID="f8acfea0-a4d3-4673-9fc6-e48a94dfbba4" Nov 02 22:47:49.128234 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:49.128195 2578 generic.go:356] "Generic (PLEG): container finished" podID="ca56f427-3734-48ec-b7ce-100daeb7b3bb" containerID="b2d264c1057858855976df1e2b843c740e925d38f9a8de51af78738734e48086" exitCode=0 Nov 02 22:47:49.129018 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:49.128268 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpqcq" event={"ID":"ca56f427-3734-48ec-b7ce-100daeb7b3bb","Type":"ContainerDied","Data":"b2d264c1057858855976df1e2b843c740e925d38f9a8de51af78738734e48086"} Nov 02 22:47:49.868388 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:49.868347 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs\") pod \"network-metrics-daemon-fqwl4\" (UID: \"5ce08d97-9151-4a82-92b3-a01a4057f046\") " pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:47:49.868612 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:49.868444 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret\") pod \"global-pull-secret-syncer-9drjv\" (UID: \"e608a3bc-a7f2-45a1-9603-d85bbbe20614\") " pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:49.868612 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:49.868528 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:49.868612 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:49.868538 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:49.868612 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:49.868597 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret podName:e608a3bc-a7f2-45a1-9603-d85bbbe20614 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:05.868584499 +0000 UTC m=+33.395229117 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret") pod "global-pull-secret-syncer-9drjv" (UID: "e608a3bc-a7f2-45a1-9603-d85bbbe20614") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:49.868612 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:49.868612 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs podName:5ce08d97-9151-4a82-92b3-a01a4057f046 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:05.8686059 +0000 UTC m=+33.395250516 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs") pod "network-metrics-daemon-fqwl4" (UID: "5ce08d97-9151-4a82-92b3-a01a4057f046") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:49.969321 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:49.969281 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxxkn\" (UniqueName: \"kubernetes.io/projected/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4-kube-api-access-kxxkn\") pod \"network-check-target-569mb\" (UID: \"f8acfea0-a4d3-4673-9fc6-e48a94dfbba4\") " pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:47:49.969524 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:49.969445 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:47:49.969524 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:49.969471 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:47:49.969524 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:49.969503 2578 projected.go:196] Error preparing data for projected volume kube-api-access-kxxkn for pod openshift-network-diagnostics/network-check-target-569mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:49.969703 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:49.969566 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4-kube-api-access-kxxkn podName:f8acfea0-a4d3-4673-9fc6-e48a94dfbba4 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:05.969548348 +0000 UTC m=+33.496192971 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kxxkn" (UniqueName: "kubernetes.io/projected/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4-kube-api-access-kxxkn") pod "network-check-target-569mb" (UID: "f8acfea0-a4d3-4673-9fc6-e48a94dfbba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:49.971498 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:49.971320 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:47:49.971498 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:49.971433 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:49.971498 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:49.971441 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:47:49.971498 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:49.971443 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-569mb" podUID="f8acfea0-a4d3-4673-9fc6-e48a94dfbba4" Nov 02 22:47:49.971741 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:49.971546 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9drjv" podUID="e608a3bc-a7f2-45a1-9603-d85bbbe20614" Nov 02 22:47:49.971741 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:49.971649 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqwl4" podUID="5ce08d97-9151-4a82-92b3-a01a4057f046" Nov 02 22:47:51.971353 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:51.971156 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:51.971809 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:51.971213 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:47:51.971809 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:51.971432 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9drjv" podUID="e608a3bc-a7f2-45a1-9603-d85bbbe20614" Nov 02 22:47:51.971809 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:51.971222 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:47:51.971809 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:51.971518 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-569mb" podUID="f8acfea0-a4d3-4673-9fc6-e48a94dfbba4" Nov 02 22:47:51.971809 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:51.971578 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqwl4" podUID="5ce08d97-9151-4a82-92b3-a01a4057f046" Nov 02 22:47:52.138421 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:52.138220 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" event={"ID":"0e82120d-ed55-489b-a3d5-a553ac1b1678","Type":"ContainerStarted","Data":"8d05f01434319e70b661916a5a0aa58ec5144e2e16ba603662a613c014486869"} Nov 02 22:47:52.138585 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:52.138471 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" event={"ID":"0e82120d-ed55-489b-a3d5-a553ac1b1678","Type":"ContainerStarted","Data":"ac9aa65e075bebfdbad3ee6c0aa36a22446e19c0e890cf9ce27788de00892cac"} Nov 02 22:47:52.138585 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:52.138533 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" event={"ID":"0e82120d-ed55-489b-a3d5-a553ac1b1678","Type":"ContainerStarted","Data":"0d9334a31d74c8d3b8095b4699a7a01c6ac5bece1bd85d6a42c02560d483cd75"} Nov 02 22:47:52.138585 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:52.138564 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" event={"ID":"0e82120d-ed55-489b-a3d5-a553ac1b1678","Type":"ContainerStarted","Data":"ba617c1a7d7d99f8ed0db0a63f56567dd93a937a54394306ca84d64a9a9c6971"} Nov 02 22:47:52.138724 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:52.138592 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" event={"ID":"0e82120d-ed55-489b-a3d5-a553ac1b1678","Type":"ContainerStarted","Data":"3708483fefbe361da6b62cebebfc9f2269d88b5e0f26cf7ad273a7fa32c0a446"} Nov 02 22:47:52.138724 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:52.138626 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" event={"ID":"0e82120d-ed55-489b-a3d5-a553ac1b1678","Type":"ContainerStarted","Data":"bb63d08fc660c56c2fd204efc49036b34ba50a58019acb3d79db199b3ba5f3fe"} Nov 02 22:47:52.141035 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:52.141006 2578 generic.go:356] "Generic (PLEG): container finished" podID="ca56f427-3734-48ec-b7ce-100daeb7b3bb" containerID="1eea70647448f7afcd915c40434937e611db4aa6cd456e23330f2e2f4476035a" exitCode=0 Nov 02 22:47:52.141172 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:52.141068 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpqcq" event={"ID":"ca56f427-3734-48ec-b7ce-100daeb7b3bb","Type":"ContainerDied","Data":"1eea70647448f7afcd915c40434937e611db4aa6cd456e23330f2e2f4476035a"} Nov 02 22:47:53.146205 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:53.146165 2578 generic.go:356] "Generic (PLEG): container finished" podID="ca56f427-3734-48ec-b7ce-100daeb7b3bb" containerID="4635195381cdb42ab8c319701accac85d0deeb3bbc6a71f0eb2e8f161ec03dc9" exitCode=0 Nov 02 22:47:53.146628 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:53.146215 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpqcq" event={"ID":"ca56f427-3734-48ec-b7ce-100daeb7b3bb","Type":"ContainerDied","Data":"4635195381cdb42ab8c319701accac85d0deeb3bbc6a71f0eb2e8f161ec03dc9"} Nov 02 22:47:53.971573 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:53.971465 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:53.971724 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:53.971465 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:47:53.971724 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:53.971586 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9drjv" podUID="e608a3bc-a7f2-45a1-9603-d85bbbe20614" Nov 02 22:47:53.971724 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:53.971465 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:47:53.971829 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:53.971653 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-569mb" podUID="f8acfea0-a4d3-4673-9fc6-e48a94dfbba4" Nov 02 22:47:53.971829 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:53.971735 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqwl4" podUID="5ce08d97-9151-4a82-92b3-a01a4057f046" Nov 02 22:47:54.151886 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:54.151852 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" event={"ID":"0e82120d-ed55-489b-a3d5-a553ac1b1678","Type":"ContainerStarted","Data":"468bd717888cccc2a1ebb1d67a2fd6929daf11982adacc87e6fa7db7475c056c"} Nov 02 22:47:54.155212 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:54.155183 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpqcq" event={"ID":"ca56f427-3734-48ec-b7ce-100daeb7b3bb","Type":"ContainerStarted","Data":"7d65cb63c282cf1097165bb4ed871d6b5ca82c0193ee8e4bd5fbf2d61eabfaea"} Nov 02 22:47:54.276382 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:54.276271 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lpqcq" podStartSLOduration=8.219415346 podStartE2EDuration="21.276258429s" totalImagesPullingTime="13.056843083s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:54.275736993 +0000 UTC m=+21.802381641" watchObservedRunningTime="2026-11-02 22:47:54.276258429 +0000 UTC m=+21.802903054" Nov 02 22:47:55.971984 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:55.971881 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:47:55.971984 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:55.971900 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:55.972746 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:55.972018 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqwl4" podUID="5ce08d97-9151-4a82-92b3-a01a4057f046" Nov 02 22:47:55.972746 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:55.972066 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:47:55.972746 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:55.972164 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9drjv" podUID="e608a3bc-a7f2-45a1-9603-d85bbbe20614" Nov 02 22:47:55.972746 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:55.972241 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-569mb" podUID="f8acfea0-a4d3-4673-9fc6-e48a94dfbba4" Nov 02 22:47:56.162722 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:56.162676 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" event={"ID":"0e82120d-ed55-489b-a3d5-a553ac1b1678","Type":"ContainerStarted","Data":"32dfa9239f3ddbd85e95d850520653e9458590a1fde505381600249958883d92"} Nov 02 22:47:56.162929 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:56.162907 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:56.179505 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:56.179460 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:56.439241 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:56.439182 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" podStartSLOduration=6.848487096 podStartE2EDuration="23.439167345s" totalImagesPullingTime="16.590680249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:56.413708748 +0000 UTC m=+23.940353383" watchObservedRunningTime="2026-11-02 22:47:56.439167345 +0000 UTC m=+23.965811970" Nov 02 22:47:57.165116 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:57.165087 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:57.165543 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:57.165153 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:57.180716 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:57.180692 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:47:57.971234 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:57.971197 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:47:57.971416 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:57.971208 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:57.971416 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:57.971317 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-569mb" podUID="f8acfea0-a4d3-4673-9fc6-e48a94dfbba4" Nov 02 22:47:57.971416 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:57.971205 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:47:57.971558 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:57.971384 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9drjv" podUID="e608a3bc-a7f2-45a1-9603-d85bbbe20614" Nov 02 22:47:57.971558 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:57.971519 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqwl4" podUID="5ce08d97-9151-4a82-92b3-a01a4057f046" Nov 02 22:47:58.510188 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:58.510151 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9drjv"] Nov 02 22:47:58.510651 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:58.510307 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:58.510651 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:58.510394 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9drjv" podUID="e608a3bc-a7f2-45a1-9603-d85bbbe20614" Nov 02 22:47:58.513197 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:58.513167 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-569mb"] Nov 02 22:47:58.513350 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:58.513279 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:47:58.513428 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:58.513374 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-569mb" podUID="f8acfea0-a4d3-4673-9fc6-e48a94dfbba4" Nov 02 22:47:58.514211 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:58.514187 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fqwl4"] Nov 02 22:47:58.514341 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:58.514271 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:47:58.514406 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:58.514351 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqwl4" podUID="5ce08d97-9151-4a82-92b3-a01a4057f046" Nov 02 22:47:59.972198 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:59.972160 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:47:59.972674 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:59.972209 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:47:59.972674 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:47:59.972265 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:47:59.972674 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:59.972276 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-569mb" podUID="f8acfea0-a4d3-4673-9fc6-e48a94dfbba4" Nov 02 22:47:59.972674 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:59.972385 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqwl4" podUID="5ce08d97-9151-4a82-92b3-a01a4057f046" Nov 02 22:47:59.972674 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:47:59.972473 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9drjv" podUID="e608a3bc-a7f2-45a1-9603-d85bbbe20614" Nov 02 22:48:01.971882 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:01.971837 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:48:01.972432 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:01.971837 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:48:01.972432 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:48:01.971978 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9drjv" podUID="e608a3bc-a7f2-45a1-9603-d85bbbe20614" Nov 02 22:48:01.972432 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:01.971854 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:48:01.972432 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:48:01.972056 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-569mb" podUID="f8acfea0-a4d3-4673-9fc6-e48a94dfbba4" Nov 02 22:48:01.972432 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:48:01.972153 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqwl4" podUID="5ce08d97-9151-4a82-92b3-a01a4057f046" Nov 02 22:48:03.971539 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:03.971497 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:48:03.971539 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:03.971500 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:48:03.972072 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:03.971519 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:48:03.972072 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:48:03.971653 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqwl4" podUID="5ce08d97-9151-4a82-92b3-a01a4057f046" Nov 02 22:48:03.972072 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:48:03.971690 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-569mb" podUID="f8acfea0-a4d3-4673-9fc6-e48a94dfbba4" Nov 02 22:48:03.972072 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:48:03.971747 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9drjv" podUID="e608a3bc-a7f2-45a1-9603-d85bbbe20614" Nov 02 22:48:04.788763 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:04.788730 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeReady" Nov 02 22:48:04.788974 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:04.788862 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 22:48:05.885755 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:05.885716 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs\") pod \"network-metrics-daemon-fqwl4\" (UID: \"5ce08d97-9151-4a82-92b3-a01a4057f046\") " pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:48:05.886147 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:05.885785 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret\") pod \"global-pull-secret-syncer-9drjv\" (UID: \"e608a3bc-a7f2-45a1-9603-d85bbbe20614\") " pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:48:05.886147 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:48:05.885880 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:48:05.886147 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:48:05.885880 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:48:05.886147 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:48:05.885938 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs podName:5ce08d97-9151-4a82-92b3-a01a4057f046 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:37.885923542 +0000 UTC m=+65.412568149 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs") pod "network-metrics-daemon-fqwl4" (UID: "5ce08d97-9151-4a82-92b3-a01a4057f046") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:48:05.886147 ip-10-0-135-250 kubenswrapper[2578]: E1102 22:48:05.885952 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret podName:e608a3bc-a7f2-45a1-9603-d85bbbe20614 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:37.885945869 +0000 UTC m=+65.412590473 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret") pod "global-pull-secret-syncer-9drjv" (UID: "e608a3bc-a7f2-45a1-9603-d85bbbe20614") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:48:05.971499 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:05.971454 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:48:05.971667 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:05.971464 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:48:05.971667 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:05.971471 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:48:05.986082 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:05.986055 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxxkn\" (UniqueName: \"kubernetes.io/projected/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4-kube-api-access-kxxkn\") pod \"network-check-target-569mb\" (UID: \"f8acfea0-a4d3-4673-9fc6-e48a94dfbba4\") " pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:48:06.023968 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:06.023931 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 22:48:06.046234 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:06.046201 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 22:48:06.050217 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:06.050195 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxxkn\" (UniqueName: \"kubernetes.io/projected/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4-kube-api-access-kxxkn\") pod \"network-check-target-569mb\" (UID: \"f8acfea0-a4d3-4673-9fc6-e48a94dfbba4\") " pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:48:06.073739 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:06.073706 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wk6k8\"" Nov 02 22:48:06.073952 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:06.073773 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zfnv8\"" Nov 02 22:48:06.073952 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:06.073794 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 22:48:06.073952 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:06.073817 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r8wwr\"" Nov 02 22:48:06.073952 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:06.073938 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 22:48:06.083063 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:06.083038 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:48:06.527841 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:06.527767 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-569mb"] Nov 02 22:48:06.531309 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:48:06.531273 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8acfea0_a4d3_4673_9fc6_e48a94dfbba4.slice/crio-9887e0ad9b044eb9cca02058d29958f530800fbc5768f2445a8c7f229b674f76 WatchSource:0}: Error finding container 9887e0ad9b044eb9cca02058d29958f530800fbc5768f2445a8c7f229b674f76: Status 404 returned error can't find the container with id 9887e0ad9b044eb9cca02058d29958f530800fbc5768f2445a8c7f229b674f76 Nov 02 22:48:07.187643 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:07.187597 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-569mb" event={"ID":"f8acfea0-a4d3-4673-9fc6-e48a94dfbba4","Type":"ContainerStarted","Data":"9887e0ad9b044eb9cca02058d29958f530800fbc5768f2445a8c7f229b674f76"} Nov 02 22:48:11.199284 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:11.199248 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-569mb" event={"ID":"f8acfea0-a4d3-4673-9fc6-e48a94dfbba4","Type":"ContainerStarted","Data":"f315c8fe1283e86289ac9fd86626c02c934bf1195a788e2e707177608bc71833"} Nov 02 22:48:11.199741 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:11.199384 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:48:11.313302 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:11.313243 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-569mb" podStartSLOduration=34.605257558 podStartE2EDuration="38.313223647s" totalImagesPullingTime="3.707966089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:11.312866119 +0000 UTC m=+38.839510740" watchObservedRunningTime="2026-11-02 22:48:11.313223647 +0000 UTC m=+38.839868272" Nov 02 22:48:12.288138 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:12.288113 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l5tth_d05df5bf-8c7c-4c79-afa6-96fc7d3689e4/dns-node-resolver/0.log" Nov 02 22:48:12.915268 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:12.915239 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dsp96_73682abc-fb01-4ac2-a073-fcffad51a512/node-ca/0.log" Nov 02 22:48:26.107845 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.107811 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8qzjx"] Nov 02 22:48:26.109531 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.109511 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vtndb"] Nov 02 22:48:26.109604 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.109584 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:26.111276 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.111248 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vtndb" Nov 02 22:48:26.131336 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.131306 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/db5f685d-3887-4e83-99f8-7538f184224f-crio-socket\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:26.131527 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.131342 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr6fc\" (UniqueName: \"kubernetes.io/projected/db5f685d-3887-4e83-99f8-7538f184224f-kube-api-access-tr6fc\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:26.131527 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.131427 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/db5f685d-3887-4e83-99f8-7538f184224f-data-volume\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:26.131527 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.131462 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db5f685d-3887-4e83-99f8-7538f184224f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:26.131698 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.131578 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl98f\" (UniqueName: \"kubernetes.io/projected/1322aa24-24c5-4a2e-b767-9bda95b4daaa-kube-api-access-rl98f\") pod \"ingress-canary-vtndb\" (UID: \"1322aa24-24c5-4a2e-b767-9bda95b4daaa\") " pod="openshift-ingress-canary/ingress-canary-vtndb" Nov 02 22:48:26.131698 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.131625 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db5f685d-3887-4e83-99f8-7538f184224f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:26.131698 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.131651 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1322aa24-24c5-4a2e-b767-9bda95b4daaa-cert\") pod \"ingress-canary-vtndb\" (UID: \"1322aa24-24c5-4a2e-b767-9bda95b4daaa\") " pod="openshift-ingress-canary/ingress-canary-vtndb" Nov 02 22:48:26.160525 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.160495 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8qzjx"] Nov 02 22:48:26.161427 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.161403 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 22:48:26.161573 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.161403 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 22:48:26.161573 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.161511 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 22:48:26.187801 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.187773 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 22:48:26.187985 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.187824 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5z48j\"" Nov 02 22:48:26.209163 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.209132 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 22:48:26.209342 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.209300 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 22:48:26.210045 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.210027 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vtndb"] Nov 02 22:48:26.211255 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.211236 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mzgfc"] Nov 02 22:48:26.212906 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.212889 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8bdc9bc8d-tcg5t"] Nov 02 22:48:26.213047 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.213030 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mzgfc" Nov 02 22:48:26.214575 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.214557 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.232115 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.232053 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/db5f685d-3887-4e83-99f8-7538f184224f-crio-socket\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:26.232272 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.232128 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0121629-a394-4f12-a90c-2f4246436bc5-registry-tls\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.232272 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.232148 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mh6p\" (UniqueName: \"kubernetes.io/projected/d0121629-a394-4f12-a90c-2f4246436bc5-kube-api-access-5mh6p\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.232272 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.232162 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/db5f685d-3887-4e83-99f8-7538f184224f-crio-socket\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:26.232272 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.232166 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr6fc\" (UniqueName: \"kubernetes.io/projected/db5f685d-3887-4e83-99f8-7538f184224f-kube-api-access-tr6fc\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:26.232272 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.232216 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0121629-a394-4f12-a90c-2f4246436bc5-registry-certificates\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.232439 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.232321 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0121629-a394-4f12-a90c-2f4246436bc5-bound-sa-token\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.232439 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.232365 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/db5f685d-3887-4e83-99f8-7538f184224f-data-volume\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:26.232523 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.232500 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db5f685d-3887-4e83-99f8-7538f184224f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:26.232557 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.232521 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl98f\" (UniqueName: \"kubernetes.io/projected/1322aa24-24c5-4a2e-b767-9bda95b4daaa-kube-api-access-rl98f\") pod \"ingress-canary-vtndb\" (UID: \"1322aa24-24c5-4a2e-b767-9bda95b4daaa\") " pod="openshift-ingress-canary/ingress-canary-vtndb" Nov 02 22:48:26.232557 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.232542 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0121629-a394-4f12-a90c-2f4246436bc5-ca-trust-extracted\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.232650 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.232561 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db5f685d-3887-4e83-99f8-7538f184224f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:26.232722 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.232703 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/db5f685d-3887-4e83-99f8-7538f184224f-data-volume\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:26.233145 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.233126 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db5f685d-3887-4e83-99f8-7538f184224f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:26.233217 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.233161 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1322aa24-24c5-4a2e-b767-9bda95b4daaa-cert\") pod \"ingress-canary-vtndb\" (UID: \"1322aa24-24c5-4a2e-b767-9bda95b4daaa\") " pod="openshift-ingress-canary/ingress-canary-vtndb" Nov 02 22:48:26.233217 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.233195 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ed00fd-8810-4f2d-868e-6ab2edcb860e-tmp-dir\") pod \"dns-default-mzgfc\" (UID: \"38ed00fd-8810-4f2d-868e-6ab2edcb860e\") " pod="openshift-dns/dns-default-mzgfc" Nov 02 22:48:26.233217 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.233212 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0121629-a394-4f12-a90c-2f4246436bc5-image-registry-private-configuration\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.233327 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.233230 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38ed00fd-8810-4f2d-868e-6ab2edcb860e-config-volume\") pod \"dns-default-mzgfc\" (UID: \"38ed00fd-8810-4f2d-868e-6ab2edcb860e\") " pod="openshift-dns/dns-default-mzgfc" Nov 02 22:48:26.233327 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.233247 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38ed00fd-8810-4f2d-868e-6ab2edcb860e-metrics-tls\") pod \"dns-default-mzgfc\" (UID: \"38ed00fd-8810-4f2d-868e-6ab2edcb860e\") " pod="openshift-dns/dns-default-mzgfc" Nov 02 22:48:26.233327 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.233278 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0121629-a394-4f12-a90c-2f4246436bc5-trusted-ca\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.233327 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.233313 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0121629-a394-4f12-a90c-2f4246436bc5-installation-pull-secrets\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.233454 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.233421 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6655\" (UniqueName: \"kubernetes.io/projected/38ed00fd-8810-4f2d-868e-6ab2edcb860e-kube-api-access-t6655\") pod \"dns-default-mzgfc\" (UID: \"38ed00fd-8810-4f2d-868e-6ab2edcb860e\") " pod="openshift-dns/dns-default-mzgfc" Nov 02 22:48:26.234996 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.234971 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db5f685d-3887-4e83-99f8-7538f184224f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:26.261964 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.261929 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mzgfc"] Nov 02 22:48:26.265970 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.265940 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8bdc9bc8d-tcg5t"] Nov 02 22:48:26.268185 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.268147 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-cz5vj\"" Nov 02 22:48:26.268332 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.268201 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 22:48:26.268664 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.268642 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 22:48:26.268771 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.268690 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 22:48:26.268835 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.268811 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9g8qt\"" Nov 02 22:48:26.268835 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.268817 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 22:48:26.268952 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.268858 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 22:48:26.268952 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.268890 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 22:48:26.275576 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.275550 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1322aa24-24c5-4a2e-b767-9bda95b4daaa-cert\") pod \"ingress-canary-vtndb\" (UID: \"1322aa24-24c5-4a2e-b767-9bda95b4daaa\") " pod="openshift-ingress-canary/ingress-canary-vtndb" Nov 02 22:48:26.318521 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.318475 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qf97t\"" Nov 02 22:48:26.318708 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.318694 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 22:48:26.321290 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.321265 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr6fc\" (UniqueName: \"kubernetes.io/projected/db5f685d-3887-4e83-99f8-7538f184224f-kube-api-access-tr6fc\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:26.321425 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.321346 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl98f\" (UniqueName: \"kubernetes.io/projected/1322aa24-24c5-4a2e-b767-9bda95b4daaa-kube-api-access-rl98f\") pod \"ingress-canary-vtndb\" (UID: \"1322aa24-24c5-4a2e-b767-9bda95b4daaa\") " pod="openshift-ingress-canary/ingress-canary-vtndb" Nov 02 22:48:26.334407 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.334371 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ed00fd-8810-4f2d-868e-6ab2edcb860e-tmp-dir\") pod \"dns-default-mzgfc\" (UID: \"38ed00fd-8810-4f2d-868e-6ab2edcb860e\") " pod="openshift-dns/dns-default-mzgfc" Nov 02 22:48:26.334407 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.334410 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0121629-a394-4f12-a90c-2f4246436bc5-image-registry-private-configuration\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.334649 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.334427 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38ed00fd-8810-4f2d-868e-6ab2edcb860e-config-volume\") pod \"dns-default-mzgfc\" (UID: \"38ed00fd-8810-4f2d-868e-6ab2edcb860e\") " pod="openshift-dns/dns-default-mzgfc" Nov 02 22:48:26.334649 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.334444 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38ed00fd-8810-4f2d-868e-6ab2edcb860e-metrics-tls\") pod \"dns-default-mzgfc\" (UID: \"38ed00fd-8810-4f2d-868e-6ab2edcb860e\") " pod="openshift-dns/dns-default-mzgfc" Nov 02 22:48:26.334649 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.334462 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0121629-a394-4f12-a90c-2f4246436bc5-trusted-ca\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.334649 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.334497 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0121629-a394-4f12-a90c-2f4246436bc5-installation-pull-secrets\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.334830 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.334645 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6655\" (UniqueName: \"kubernetes.io/projected/38ed00fd-8810-4f2d-868e-6ab2edcb860e-kube-api-access-t6655\") pod \"dns-default-mzgfc\" (UID: \"38ed00fd-8810-4f2d-868e-6ab2edcb860e\") " pod="openshift-dns/dns-default-mzgfc" Nov 02 22:48:26.334830 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.334707 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0121629-a394-4f12-a90c-2f4246436bc5-registry-tls\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.334927 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.334853 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mh6p\" (UniqueName: \"kubernetes.io/projected/d0121629-a394-4f12-a90c-2f4246436bc5-kube-api-access-5mh6p\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.334927 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.334878 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ed00fd-8810-4f2d-868e-6ab2edcb860e-tmp-dir\") pod \"dns-default-mzgfc\" (UID: \"38ed00fd-8810-4f2d-868e-6ab2edcb860e\") " pod="openshift-dns/dns-default-mzgfc" Nov 02 22:48:26.334927 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.334887 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0121629-a394-4f12-a90c-2f4246436bc5-registry-certificates\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.335066 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.334933 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0121629-a394-4f12-a90c-2f4246436bc5-bound-sa-token\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.335446 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.335418 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0121629-a394-4f12-a90c-2f4246436bc5-ca-trust-extracted\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.335622 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.335600 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0121629-a394-4f12-a90c-2f4246436bc5-trusted-ca\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.335814 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.335793 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0121629-a394-4f12-a90c-2f4246436bc5-ca-trust-extracted\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.336050 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.336028 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0121629-a394-4f12-a90c-2f4246436bc5-registry-certificates\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.336247 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.336225 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38ed00fd-8810-4f2d-868e-6ab2edcb860e-config-volume\") pod \"dns-default-mzgfc\" (UID: \"38ed00fd-8810-4f2d-868e-6ab2edcb860e\") " pod="openshift-dns/dns-default-mzgfc" Nov 02 22:48:26.337375 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.337348 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0121629-a394-4f12-a90c-2f4246436bc5-image-registry-private-configuration\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.337541 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.337523 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38ed00fd-8810-4f2d-868e-6ab2edcb860e-metrics-tls\") pod \"dns-default-mzgfc\" (UID: \"38ed00fd-8810-4f2d-868e-6ab2edcb860e\") " pod="openshift-dns/dns-default-mzgfc" Nov 02 22:48:26.337624 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.337523 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0121629-a394-4f12-a90c-2f4246436bc5-registry-tls\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.337624 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.337575 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0121629-a394-4f12-a90c-2f4246436bc5-installation-pull-secrets\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.397949 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.397867 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0121629-a394-4f12-a90c-2f4246436bc5-bound-sa-token\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.397949 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.397877 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6655\" (UniqueName: \"kubernetes.io/projected/38ed00fd-8810-4f2d-868e-6ab2edcb860e-kube-api-access-t6655\") pod \"dns-default-mzgfc\" (UID: \"38ed00fd-8810-4f2d-868e-6ab2edcb860e\") " pod="openshift-dns/dns-default-mzgfc" Nov 02 22:48:26.398326 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.398301 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mh6p\" (UniqueName: \"kubernetes.io/projected/d0121629-a394-4f12-a90c-2f4246436bc5-kube-api-access-5mh6p\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.423937 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.423906 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:26.424969 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.424947 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vtndb" Nov 02 22:48:26.525032 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.524997 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mzgfc" Nov 02 22:48:26.529244 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.529222 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:26.851526 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.851471 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vtndb"] Nov 02 22:48:26.854186 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.854154 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8qzjx"] Nov 02 22:48:26.854833 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:48:26.854804 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1322aa24_24c5_4a2e_b767_9bda95b4daaa.slice/crio-31f0641478a464484c9f4d8b85a2165de817c41bc124000b33749a28401bba05 WatchSource:0}: Error finding container 31f0641478a464484c9f4d8b85a2165de817c41bc124000b33749a28401bba05: Status 404 returned error can't find the container with id 31f0641478a464484c9f4d8b85a2165de817c41bc124000b33749a28401bba05 Nov 02 22:48:26.899566 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:48:26.899534 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb5f685d_3887_4e83_99f8_7538f184224f.slice/crio-9e95567ad1efeb49460dea7e0e951b3c0c82b7739e5d86b3220a03cb0f25a531 WatchSource:0}: Error finding container 9e95567ad1efeb49460dea7e0e951b3c0c82b7739e5d86b3220a03cb0f25a531: Status 404 returned error can't find the container with id 9e95567ad1efeb49460dea7e0e951b3c0c82b7739e5d86b3220a03cb0f25a531 Nov 02 22:48:26.908654 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.908631 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mzgfc"] Nov 02 22:48:26.912962 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:26.912881 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8bdc9bc8d-tcg5t"] Nov 02 22:48:26.913068 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:48:26.913041 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ed00fd_8810_4f2d_868e_6ab2edcb860e.slice/crio-b113915b35fee03e088cb22493cd9621f8a66cce5eb9036eb4ccde329bbf16ff WatchSource:0}: Error finding container b113915b35fee03e088cb22493cd9621f8a66cce5eb9036eb4ccde329bbf16ff: Status 404 returned error can't find the container with id b113915b35fee03e088cb22493cd9621f8a66cce5eb9036eb4ccde329bbf16ff Nov 02 22:48:26.917904 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:48:26.917879 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0121629_a394_4f12_a90c_2f4246436bc5.slice/crio-00d4ccbee98a6deedb9d504bf767e5a7a9cf38c0c1f85f8ba00911ad35ff1ab6 WatchSource:0}: Error finding container 00d4ccbee98a6deedb9d504bf767e5a7a9cf38c0c1f85f8ba00911ad35ff1ab6: Status 404 returned error can't find the container with id 00d4ccbee98a6deedb9d504bf767e5a7a9cf38c0c1f85f8ba00911ad35ff1ab6 Nov 02 22:48:27.237810 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:27.237706 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" event={"ID":"d0121629-a394-4f12-a90c-2f4246436bc5","Type":"ContainerStarted","Data":"4661ce24fc7c21b9abe4401054fb076b1d8891b3a155e371d70cf9bd6dd7e681"} Nov 02 22:48:27.237810 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:27.237782 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:27.237810 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:27.237803 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" event={"ID":"d0121629-a394-4f12-a90c-2f4246436bc5","Type":"ContainerStarted","Data":"00d4ccbee98a6deedb9d504bf767e5a7a9cf38c0c1f85f8ba00911ad35ff1ab6"} Nov 02 22:48:27.239285 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:27.239251 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vtndb" event={"ID":"1322aa24-24c5-4a2e-b767-9bda95b4daaa","Type":"ContainerStarted","Data":"31f0641478a464484c9f4d8b85a2165de817c41bc124000b33749a28401bba05"} Nov 02 22:48:27.241081 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:27.241054 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8qzjx" event={"ID":"db5f685d-3887-4e83-99f8-7538f184224f","Type":"ContainerStarted","Data":"59482adc8b0166d000417b124c151c3a2d442a59011b90b5dd90c428fd3efc81"} Nov 02 22:48:27.241205 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:27.241111 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8qzjx" event={"ID":"db5f685d-3887-4e83-99f8-7538f184224f","Type":"ContainerStarted","Data":"9e95567ad1efeb49460dea7e0e951b3c0c82b7739e5d86b3220a03cb0f25a531"} Nov 02 22:48:27.242317 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:27.242279 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mzgfc" event={"ID":"38ed00fd-8810-4f2d-868e-6ab2edcb860e","Type":"ContainerStarted","Data":"b113915b35fee03e088cb22493cd9621f8a66cce5eb9036eb4ccde329bbf16ff"} Nov 02 22:48:27.356143 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:27.356091 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" podStartSLOduration=1.35607621 podStartE2EDuration="1.35607621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:27.355669779 +0000 UTC m=+54.882314405" watchObservedRunningTime="2026-11-02 22:48:27.35607621 +0000 UTC m=+54.882720835" Nov 02 22:48:28.248315 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:28.248276 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8qzjx" event={"ID":"db5f685d-3887-4e83-99f8-7538f184224f","Type":"ContainerStarted","Data":"ada166d25f00449bf51c4eb42ca1d310cf39b216ef6ad762bed440253921a7b6"} Nov 02 22:48:29.185920 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:29.185888 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:30.257016 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:30.256977 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vtndb" event={"ID":"1322aa24-24c5-4a2e-b767-9bda95b4daaa","Type":"ContainerStarted","Data":"f8cd243dcb07314d98974414f391ec399b9d9f5050bd69fcdb4937de3b3db8a7"} Nov 02 22:48:30.259118 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:30.259088 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mzgfc" event={"ID":"38ed00fd-8810-4f2d-868e-6ab2edcb860e","Type":"ContainerStarted","Data":"998baf4ddc9ab363a65a5139d9888481cba423b682c052d174a37748c6d870a5"} Nov 02 22:48:30.259257 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:30.259145 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mzgfc" event={"ID":"38ed00fd-8810-4f2d-868e-6ab2edcb860e","Type":"ContainerStarted","Data":"77549ce50525299886741082e7aabb6e9bf0b7b831f487998e21c8a2f586ccdd"} Nov 02 22:48:30.259257 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:30.259177 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mzgfc" Nov 02 22:48:30.372898 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:30.372846 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vtndb" podStartSLOduration=1.134186858 podStartE2EDuration="4.37282951s" totalImagesPullingTime="3.238642652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:30.371862116 +0000 UTC m=+57.898506753" watchObservedRunningTime="2026-11-02 22:48:30.37282951 +0000 UTC m=+57.899474131" Nov 02 22:48:30.488843 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:30.488783 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-mzgfc" podStartSLOduration=2.031827875 podStartE2EDuration="4.488767269s" totalImagesPullingTime="2.456939394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:30.488013404 +0000 UTC m=+58.014658030" watchObservedRunningTime="2026-11-02 22:48:30.488767269 +0000 UTC m=+58.015411893" Nov 02 22:48:31.265890 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:31.265843 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8qzjx" event={"ID":"db5f685d-3887-4e83-99f8-7538f184224f","Type":"ContainerStarted","Data":"08800dbed740bd2ea20b09f6e9990714646a969ee24133aa194d26425071a5bd"} Nov 02 22:48:31.383764 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:31.383709 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8qzjx" podStartSLOduration=2.127984247 podStartE2EDuration="5.38369167s" totalImagesPullingTime="3.255707423s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:48:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:31.382785865 +0000 UTC m=+58.909430492" watchObservedRunningTime="2026-11-02 22:48:31.38369167 +0000 UTC m=+58.910336337" Nov 02 22:48:32.773965 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:32.773929 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8lzdt"] Nov 02 22:48:32.776156 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:32.776137 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8lzdt" Nov 02 22:48:32.828312 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:32.828278 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 02 22:48:32.877677 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:32.877642 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bd5jt\"" Nov 02 22:48:32.878622 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:32.878603 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8lzdt"] Nov 02 22:48:32.888194 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:32.888160 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ad69acdc-e65a-4aad-ac33-354f6ea07401-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-8lzdt\" (UID: \"ad69acdc-e65a-4aad-ac33-354f6ea07401\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8lzdt" Nov 02 22:48:32.988631 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:32.988585 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ad69acdc-e65a-4aad-ac33-354f6ea07401-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-8lzdt\" (UID: \"ad69acdc-e65a-4aad-ac33-354f6ea07401\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8lzdt" Nov 02 22:48:32.991232 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:32.991205 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ad69acdc-e65a-4aad-ac33-354f6ea07401-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-8lzdt\" (UID: \"ad69acdc-e65a-4aad-ac33-354f6ea07401\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8lzdt" Nov 02 22:48:33.085462 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:33.085426 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8lzdt" Nov 02 22:48:33.453357 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:33.453149 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8lzdt"] Nov 02 22:48:33.455831 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:48:33.455790 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad69acdc_e65a_4aad_ac33_354f6ea07401.slice/crio-b77d42eb9ea696f91f01f98fa13c0571e8ecbc3d8d5e0039236a7fe32d46c502 WatchSource:0}: Error finding container b77d42eb9ea696f91f01f98fa13c0571e8ecbc3d8d5e0039236a7fe32d46c502: Status 404 returned error can't find the container with id b77d42eb9ea696f91f01f98fa13c0571e8ecbc3d8d5e0039236a7fe32d46c502 Nov 02 22:48:34.276000 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:34.275965 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8lzdt" event={"ID":"ad69acdc-e65a-4aad-ac33-354f6ea07401","Type":"ContainerStarted","Data":"b77d42eb9ea696f91f01f98fa13c0571e8ecbc3d8d5e0039236a7fe32d46c502"} Nov 02 22:48:35.281081 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:35.280993 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8lzdt" event={"ID":"ad69acdc-e65a-4aad-ac33-354f6ea07401","Type":"ContainerStarted","Data":"1cf10a93e691e6a53303f3cfe22b5e94ad9d725bb3046d2e458fb2d87eeb5341"} Nov 02 22:48:35.281532 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:35.281227 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8lzdt" Nov 02 22:48:35.285746 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:35.285726 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8lzdt" Nov 02 22:48:35.396968 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:35.396907 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8lzdt" podStartSLOduration=1.840673158 podStartE2EDuration="3.396894311s" totalImagesPullingTime="1.556221153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:35.396002805 +0000 UTC m=+62.922647430" watchObservedRunningTime="2026-11-02 22:48:35.396894311 +0000 UTC m=+62.923538937" Nov 02 22:48:36.375043 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:36.375013 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-p266m"] Nov 02 22:48:36.376822 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:36.376807 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" Nov 02 22:48:36.415619 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:36.415569 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw7mm\" (UniqueName: \"kubernetes.io/projected/94071069-300b-43de-87e3-58e31ac8ab9d-kube-api-access-pw7mm\") pod \"prometheus-operator-9cd89b74c-p266m\" (UID: \"94071069-300b-43de-87e3-58e31ac8ab9d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" Nov 02 22:48:36.415825 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:36.415645 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94071069-300b-43de-87e3-58e31ac8ab9d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-p266m\" (UID: \"94071069-300b-43de-87e3-58e31ac8ab9d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" Nov 02 22:48:36.415825 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:36.415686 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94071069-300b-43de-87e3-58e31ac8ab9d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-p266m\" (UID: \"94071069-300b-43de-87e3-58e31ac8ab9d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" Nov 02 22:48:36.415825 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:36.415783 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/94071069-300b-43de-87e3-58e31ac8ab9d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-p266m\" (UID: \"94071069-300b-43de-87e3-58e31ac8ab9d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" Nov 02 22:48:36.478169 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:36.478121 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 22:48:36.478169 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:36.478118 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 22:48:36.478393 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:36.478194 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-cqvbp\"" Nov 02 22:48:36.478393 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:36.478200 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 22:48:36.478393 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:36.478293 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 22:48:36.478393 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:36.478354 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 22:48:36.479329 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:36.479311 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-p266m"] Nov 02 22:48:36.516759 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:36.516720 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94071069-300b-43de-87e3-58e31ac8ab9d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-p266m\" (UID: \"94071069-300b-43de-87e3-58e31ac8ab9d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" Nov 02 22:48:36.516759 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:36.516756 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94071069-300b-43de-87e3-58e31ac8ab9d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-p266m\" (UID: \"94071069-300b-43de-87e3-58e31ac8ab9d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" Nov 02 22:48:36.517011 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:36.516800 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/94071069-300b-43de-87e3-58e31ac8ab9d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-p266m\" (UID: \"94071069-300b-43de-87e3-58e31ac8ab9d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" Nov 02 22:48:36.517011 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:36.516845 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw7mm\" (UniqueName: \"kubernetes.io/projected/94071069-300b-43de-87e3-58e31ac8ab9d-kube-api-access-pw7mm\") pod \"prometheus-operator-9cd89b74c-p266m\" (UID: \"94071069-300b-43de-87e3-58e31ac8ab9d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" Nov 02 22:48:36.517590 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:36.517559 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94071069-300b-43de-87e3-58e31ac8ab9d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-p266m\" (UID: \"94071069-300b-43de-87e3-58e31ac8ab9d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" Nov 02 22:48:36.519362 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:36.519342 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/94071069-300b-43de-87e3-58e31ac8ab9d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-p266m\" (UID: \"94071069-300b-43de-87e3-58e31ac8ab9d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" Nov 02 22:48:36.519454 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:36.519342 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94071069-300b-43de-87e3-58e31ac8ab9d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-p266m\" (UID: \"94071069-300b-43de-87e3-58e31ac8ab9d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" Nov 02 22:48:36.577848 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:36.577805 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw7mm\" (UniqueName: \"kubernetes.io/projected/94071069-300b-43de-87e3-58e31ac8ab9d-kube-api-access-pw7mm\") pod \"prometheus-operator-9cd89b74c-p266m\" (UID: \"94071069-300b-43de-87e3-58e31ac8ab9d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" Nov 02 22:48:36.687156 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:36.687064 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" Nov 02 22:48:37.057150 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:37.057120 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-p266m"] Nov 02 22:48:37.060320 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:48:37.060293 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94071069_300b_43de_87e3_58e31ac8ab9d.slice/crio-2c8a3a269fc77b0500375f98669c75aa984112b8933bf283336a1eafd429f027 WatchSource:0}: Error finding container 2c8a3a269fc77b0500375f98669c75aa984112b8933bf283336a1eafd429f027: Status 404 returned error can't find the container with id 2c8a3a269fc77b0500375f98669c75aa984112b8933bf283336a1eafd429f027 Nov 02 22:48:37.288908 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:37.288875 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" event={"ID":"94071069-300b-43de-87e3-58e31ac8ab9d","Type":"ContainerStarted","Data":"2c8a3a269fc77b0500375f98669c75aa984112b8933bf283336a1eafd429f027"} Nov 02 22:48:37.492526 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:37.492425 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-vtccr"] Nov 02 22:48:37.494738 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:37.494716 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-vtccr" Nov 02 22:48:37.526971 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:37.526940 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6048231-1f5f-482a-af32-228a7c75f910-tmp\") pod \"downloads-6dd789c8b7-vtccr\" (UID: \"b6048231-1f5f-482a-af32-228a7c75f910\") " pod="openshift-console/downloads-6dd789c8b7-vtccr" Nov 02 22:48:37.527138 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:37.526983 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7zxb\" (UniqueName: \"kubernetes.io/projected/b6048231-1f5f-482a-af32-228a7c75f910-kube-api-access-m7zxb\") pod \"downloads-6dd789c8b7-vtccr\" (UID: \"b6048231-1f5f-482a-af32-228a7c75f910\") " pod="openshift-console/downloads-6dd789c8b7-vtccr" Nov 02 22:48:37.550640 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:37.550612 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 22:48:37.550640 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:37.550612 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 22:48:37.550901 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:37.550859 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-467lc\"" Nov 02 22:48:37.554557 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:37.554534 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-vtccr"] Nov 02 22:48:37.627363 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:37.627324 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6048231-1f5f-482a-af32-228a7c75f910-tmp\") pod \"downloads-6dd789c8b7-vtccr\" (UID: \"b6048231-1f5f-482a-af32-228a7c75f910\") " pod="openshift-console/downloads-6dd789c8b7-vtccr" Nov 02 22:48:37.627363 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:37.627363 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7zxb\" (UniqueName: \"kubernetes.io/projected/b6048231-1f5f-482a-af32-228a7c75f910-kube-api-access-m7zxb\") pod \"downloads-6dd789c8b7-vtccr\" (UID: \"b6048231-1f5f-482a-af32-228a7c75f910\") " pod="openshift-console/downloads-6dd789c8b7-vtccr" Nov 02 22:48:37.627815 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:37.627792 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6048231-1f5f-482a-af32-228a7c75f910-tmp\") pod \"downloads-6dd789c8b7-vtccr\" (UID: \"b6048231-1f5f-482a-af32-228a7c75f910\") " pod="openshift-console/downloads-6dd789c8b7-vtccr" Nov 02 22:48:37.688231 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:37.688187 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7zxb\" (UniqueName: \"kubernetes.io/projected/b6048231-1f5f-482a-af32-228a7c75f910-kube-api-access-m7zxb\") pod \"downloads-6dd789c8b7-vtccr\" (UID: \"b6048231-1f5f-482a-af32-228a7c75f910\") " pod="openshift-console/downloads-6dd789c8b7-vtccr" Nov 02 22:48:37.806911 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:37.806855 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-vtccr" Nov 02 22:48:37.929649 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:37.929607 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret\") pod \"global-pull-secret-syncer-9drjv\" (UID: \"e608a3bc-a7f2-45a1-9603-d85bbbe20614\") " pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:48:37.929649 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:37.929652 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs\") pod \"network-metrics-daemon-fqwl4\" (UID: \"5ce08d97-9151-4a82-92b3-a01a4057f046\") " pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:48:37.980835 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:37.980797 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 22:48:37.981016 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:37.980952 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 22:48:37.992660 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:37.992626 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs\") pod \"network-metrics-daemon-fqwl4\" (UID: \"5ce08d97-9151-4a82-92b3-a01a4057f046\") " pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:48:37.992829 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:37.992737 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret\") pod \"global-pull-secret-syncer-9drjv\" (UID: \"e608a3bc-a7f2-45a1-9603-d85bbbe20614\") " pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:48:38.192850 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:38.192763 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zfnv8\"" Nov 02 22:48:38.202087 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:38.202043 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:48:38.226715 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:38.226684 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-vtccr"] Nov 02 22:48:38.231090 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:48:38.231055 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6048231_1f5f_482a_af32_228a7c75f910.slice/crio-50a8014b3378c3ec8a909389a05a913ec59d4a9b30c4a80f1efd93bdceceaf01 WatchSource:0}: Error finding container 50a8014b3378c3ec8a909389a05a913ec59d4a9b30c4a80f1efd93bdceceaf01: Status 404 returned error can't find the container with id 50a8014b3378c3ec8a909389a05a913ec59d4a9b30c4a80f1efd93bdceceaf01 Nov 02 22:48:38.238657 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:38.238625 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r8wwr\"" Nov 02 22:48:38.247664 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:38.247620 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:48:38.294857 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:38.294802 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-vtccr" event={"ID":"b6048231-1f5f-482a-af32-228a7c75f910","Type":"ContainerStarted","Data":"50a8014b3378c3ec8a909389a05a913ec59d4a9b30c4a80f1efd93bdceceaf01"} Nov 02 22:48:38.577854 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:38.577824 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9drjv"] Nov 02 22:48:38.581587 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:48:38.581553 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode608a3bc_a7f2_45a1_9603_d85bbbe20614.slice/crio-2d746a2fbc6bc62606a9cdbe5304b36eed622f947615dd7572e7fd331e4ef8cb WatchSource:0}: Error finding container 2d746a2fbc6bc62606a9cdbe5304b36eed622f947615dd7572e7fd331e4ef8cb: Status 404 returned error can't find the container with id 2d746a2fbc6bc62606a9cdbe5304b36eed622f947615dd7572e7fd331e4ef8cb Nov 02 22:48:38.636105 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:38.636071 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fqwl4"] Nov 02 22:48:38.640318 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:48:38.640275 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce08d97_9151_4a82_92b3_a01a4057f046.slice/crio-39117eeba9a2f4d86737c81b31806feef362b7e62df1d7fb6acf06b9904ba182 WatchSource:0}: Error finding container 39117eeba9a2f4d86737c81b31806feef362b7e62df1d7fb6acf06b9904ba182: Status 404 returned error can't find the container with id 39117eeba9a2f4d86737c81b31806feef362b7e62df1d7fb6acf06b9904ba182 Nov 02 22:48:39.301814 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:39.301768 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" event={"ID":"94071069-300b-43de-87e3-58e31ac8ab9d","Type":"ContainerStarted","Data":"74256d0133b43c1382cf1f3d19c24ada8f15c38f2cf30d465cce7c0bfde79915"} Nov 02 22:48:39.302005 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:39.301883 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" event={"ID":"94071069-300b-43de-87e3-58e31ac8ab9d","Type":"ContainerStarted","Data":"9a9ccc9853ea55c4f5d259826f4c296ed9d6466503aa325280d52300959bca20"} Nov 02 22:48:39.303233 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:39.303188 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fqwl4" event={"ID":"5ce08d97-9151-4a82-92b3-a01a4057f046","Type":"ContainerStarted","Data":"39117eeba9a2f4d86737c81b31806feef362b7e62df1d7fb6acf06b9904ba182"} Nov 02 22:48:39.304445 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:39.304411 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9drjv" event={"ID":"e608a3bc-a7f2-45a1-9603-d85bbbe20614","Type":"ContainerStarted","Data":"2d746a2fbc6bc62606a9cdbe5304b36eed622f947615dd7572e7fd331e4ef8cb"} Nov 02 22:48:39.417340 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:39.417278 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" podStartSLOduration=1.789962186 podStartE2EDuration="3.417263776s" totalImagesPullingTime="1.62730159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:48:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:39.416929902 +0000 UTC m=+66.943574544" watchObservedRunningTime="2026-11-02 22:48:39.417263776 +0000 UTC m=+66.943908404" Nov 02 22:48:40.215562 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:40.215533 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mzgfc_38ed00fd-8810-4f2d-868e-6ab2edcb860e/dns/0.log" Nov 02 22:48:40.268072 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:40.268040 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mzgfc" Nov 02 22:48:40.311806 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:40.311169 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fqwl4" event={"ID":"5ce08d97-9151-4a82-92b3-a01a4057f046","Type":"ContainerStarted","Data":"897a9e729b1ea5813f67626c9591af7b6b6d6c8bfdd7576fae32f9d485a97d63"} Nov 02 22:48:40.320449 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:40.320341 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mzgfc_38ed00fd-8810-4f2d-868e-6ab2edcb860e/kube-rbac-proxy/0.log" Nov 02 22:48:40.919895 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:40.919864 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l5tth_d05df5bf-8c7c-4c79-afa6-96fc7d3689e4/dns-node-resolver/0.log" Nov 02 22:48:41.316733 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:41.316646 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fqwl4" event={"ID":"5ce08d97-9151-4a82-92b3-a01a4057f046","Type":"ContainerStarted","Data":"e6cbcd1cb84875eb1f113f4098a81cc3fd070bdced3e7088909c30ea866ed2c7"} Nov 02 22:48:41.435811 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:41.435716 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fqwl4" podStartSLOduration=66.888070817 podStartE2EDuration="1m8.435701862s" totalImagesPullingTime="1.547631045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:41.433864133 +0000 UTC m=+68.960508758" watchObservedRunningTime="2026-11-02 22:48:41.435701862 +0000 UTC m=+68.962346530" Nov 02 22:48:41.919705 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:41.919673 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vtndb_1322aa24-24c5-4a2e-b767-9bda95b4daaa/serve-healthcheck-canary/0.log" Nov 02 22:48:42.206117 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:42.205564 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:48:42.875724 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:42.874498 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2"] Nov 02 22:48:42.876810 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:42.876784 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:42.924357 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:42.924327 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pw6nq"] Nov 02 22:48:42.927537 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:42.927503 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:42.934565 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:42.934520 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 22:48:42.934565 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:42.934548 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-v2s7h\"" Nov 02 22:48:42.970668 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:42.970627 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-metrics-client-ca\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:42.970833 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:42.970688 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c414822-2ac4-4cd7-844f-a1f334dba25a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:42.970833 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:42.970781 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw64f\" (UniqueName: \"kubernetes.io/projected/0c414822-2ac4-4cd7-844f-a1f334dba25a-kube-api-access-gw64f\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:42.970833 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:42.970828 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-sys\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:42.970960 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:42.970863 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-accelerators-collector-config\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:42.970960 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:42.970886 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0c414822-2ac4-4cd7-844f-a1f334dba25a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:42.970960 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:42.970910 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-textfile\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:42.970960 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:42.970926 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-wtmp\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:42.970960 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:42.970951 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77psj\" (UniqueName: \"kubernetes.io/projected/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-kube-api-access-77psj\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:42.971128 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:42.970976 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c414822-2ac4-4cd7-844f-a1f334dba25a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:42.971128 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:42.971013 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0c414822-2ac4-4cd7-844f-a1f334dba25a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:42.971128 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:42.971044 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-tls\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:42.971128 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:42.971085 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:42.971128 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:42.971110 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c414822-2ac4-4cd7-844f-a1f334dba25a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:42.971292 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:42.971141 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-root\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:42.971818 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:42.971790 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 22:48:42.979142 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:42.979111 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2"] Nov 02 22:48:42.979302 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:42.979272 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 22:48:43.026811 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.026774 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 22:48:43.027002 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.026854 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 22:48:43.027002 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.026966 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-scv9s\"" Nov 02 22:48:43.027123 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.027078 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 22:48:43.071939 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.071891 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c414822-2ac4-4cd7-844f-a1f334dba25a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:43.072103 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.071947 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw64f\" (UniqueName: \"kubernetes.io/projected/0c414822-2ac4-4cd7-844f-a1f334dba25a-kube-api-access-gw64f\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:43.072103 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.071974 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-sys\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:43.072103 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.072006 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-accelerators-collector-config\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:43.072103 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.072036 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0c414822-2ac4-4cd7-844f-a1f334dba25a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:43.072103 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.072069 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-textfile\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:43.072103 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.072094 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-wtmp\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:43.072368 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.072119 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77psj\" (UniqueName: \"kubernetes.io/projected/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-kube-api-access-77psj\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:43.072368 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.072148 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c414822-2ac4-4cd7-844f-a1f334dba25a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:43.072368 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.072189 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0c414822-2ac4-4cd7-844f-a1f334dba25a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:43.072368 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.072218 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-tls\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:43.072368 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.072256 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:43.072368 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.072280 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c414822-2ac4-4cd7-844f-a1f334dba25a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:43.072368 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.072308 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-root\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:43.072368 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.072344 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-metrics-client-ca\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:43.073042 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.073021 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-metrics-client-ca\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:43.074041 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.074014 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0c414822-2ac4-4cd7-844f-a1f334dba25a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:43.074148 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.074118 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0c414822-2ac4-4cd7-844f-a1f334dba25a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:43.074324 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.074302 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-textfile\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:43.074397 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.074346 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c414822-2ac4-4cd7-844f-a1f334dba25a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:43.074501 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.074464 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-wtmp\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:43.074623 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.074545 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-root\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:43.074623 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.074601 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-sys\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:43.074736 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.074702 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-accelerators-collector-config\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:43.076441 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.076418 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c414822-2ac4-4cd7-844f-a1f334dba25a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:43.077132 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.077102 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c414822-2ac4-4cd7-844f-a1f334dba25a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:43.077234 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.077144 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-tls\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:43.077620 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.077601 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:43.134205 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.134131 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw64f\" (UniqueName: \"kubernetes.io/projected/0c414822-2ac4-4cd7-844f-a1f334dba25a-kube-api-access-gw64f\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:43.141464 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.141435 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77psj\" (UniqueName: \"kubernetes.io/projected/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-kube-api-access-77psj\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:43.193127 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.193076 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:43.242035 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.241798 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:43.264666 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:48:43.264636 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b3abd8_5242_4bfd_bea9_9d4e0f4c4d4d.slice/crio-af5ad537206c6aee77ca429dbccb84de80c1cd0c900448f27ec20ef813a5f69d WatchSource:0}: Error finding container af5ad537206c6aee77ca429dbccb84de80c1cd0c900448f27ec20ef813a5f69d: Status 404 returned error can't find the container with id af5ad537206c6aee77ca429dbccb84de80c1cd0c900448f27ec20ef813a5f69d Nov 02 22:48:43.326994 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.326959 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pw6nq" event={"ID":"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d","Type":"ContainerStarted","Data":"af5ad537206c6aee77ca429dbccb84de80c1cd0c900448f27ec20ef813a5f69d"} Nov 02 22:48:43.625806 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:43.625767 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2"] Nov 02 22:48:43.629639 ip-10-0-135-250 kubenswrapper[2578]: W1102 22:48:43.629600 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c414822_2ac4_4cd7_844f_a1f334dba25a.slice/crio-901a3af424db738e9f6f250dd427c12d5a46087d7b5623159f66f7c229fd230f WatchSource:0}: Error finding container 901a3af424db738e9f6f250dd427c12d5a46087d7b5623159f66f7c229fd230f: Status 404 returned error can't find the container with id 901a3af424db738e9f6f250dd427c12d5a46087d7b5623159f66f7c229fd230f Nov 02 22:48:44.331859 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:44.331809 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" event={"ID":"0c414822-2ac4-4cd7-844f-a1f334dba25a","Type":"ContainerStarted","Data":"901a3af424db738e9f6f250dd427c12d5a46087d7b5623159f66f7c229fd230f"} Nov 02 22:48:44.593690 ip-10-0-135-250 kubenswrapper[2578]: I1102 22:48:44.593601 2578 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:48:44.593798 ip-10-0-135-250 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 22:48:44.602324 ip-10-0-135-250 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 22:48:44.602618 ip-10-0-135-250 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 22:48:44.602690 ip-10-0-135-250 systemd[1]: kubelet.service: Consumed 3.584s CPU time. Nov 02 22:48:44.611291 ip-10-0-135-250 systemd[1]: Starting Kubernetes Kubelet... Nov 02 22:48:44.699069 ip-10-0-135-250 kubenswrapper[5114]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:48:44.699069 ip-10-0-135-250 kubenswrapper[5114]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 22:48:44.699069 ip-10-0-135-250 kubenswrapper[5114]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:48:44.699069 ip-10-0-135-250 kubenswrapper[5114]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:48:44.701322 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701215 5114 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:48:44.701322 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701321 5114 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:48:44.701322 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701325 5114 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:48:44.701322 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701328 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:48:44.701449 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701332 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:48:44.701449 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701335 5114 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:48:44.701449 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701337 5114 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:48:44.701449 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701340 5114 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:48:44.701449 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701343 5114 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:48:44.701449 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701345 5114 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:48:44.701449 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701348 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:48:44.701449 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701350 5114 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:48:44.701449 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701354 5114 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:48:44.701449 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701357 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:48:44.701449 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701359 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:48:44.701449 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701362 5114 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:48:44.701449 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701365 5114 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:48:44.701449 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701367 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:48:44.701449 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701370 5114 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:48:44.701449 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701372 5114 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:48:44.701449 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701375 5114 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:48:44.701449 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701382 5114 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:48:44.701449 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701385 5114 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:48:44.701942 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701387 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:48:44.701942 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701390 5114 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:48:44.701942 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701392 5114 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:48:44.701942 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701395 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:48:44.701942 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701397 5114 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:48:44.701942 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701400 5114 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:48:44.701942 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701403 5114 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:48:44.701942 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701406 5114 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:48:44.701942 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701408 5114 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:48:44.701942 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701411 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:48:44.701942 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701414 5114 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:48:44.701942 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701416 5114 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:48:44.701942 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701419 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:48:44.701942 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701422 5114 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:48:44.701942 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701424 5114 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:48:44.701942 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701427 5114 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:48:44.701942 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701430 5114 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:48:44.701942 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701433 5114 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:48:44.701942 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701435 5114 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:48:44.702440 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701440 5114 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:48:44.702440 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701444 5114 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:48:44.702440 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701447 5114 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:48:44.702440 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701450 5114 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:48:44.702440 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701452 5114 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:48:44.702440 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701455 5114 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:48:44.702440 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701458 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:48:44.702440 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701460 5114 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:48:44.702440 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701463 5114 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:48:44.702440 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701466 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:48:44.702440 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701468 5114 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:48:44.702440 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701471 5114 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:48:44.702440 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701474 5114 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:48:44.702440 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701476 5114 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:48:44.702440 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701494 5114 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:48:44.702440 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701497 5114 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:48:44.702440 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701500 5114 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:48:44.702440 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701503 5114 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:48:44.702440 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701505 5114 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:48:44.702926 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701509 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:48:44.702926 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701512 5114 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:48:44.702926 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701514 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:48:44.702926 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701517 5114 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:48:44.702926 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701520 5114 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:48:44.702926 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701522 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:48:44.702926 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701525 5114 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:48:44.702926 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701528 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:48:44.702926 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701530 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:48:44.702926 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701533 5114 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:48:44.702926 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701535 5114 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:48:44.702926 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701543 5114 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:48:44.702926 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701547 5114 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:48:44.702926 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701549 5114 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:48:44.702926 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701553 5114 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:48:44.702926 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701555 5114 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:48:44.702926 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701558 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:48:44.702926 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701561 5114 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:48:44.702926 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701564 5114 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:48:44.703381 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701566 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:48:44.703381 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701569 5114 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:48:44.703381 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701572 5114 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:48:44.703381 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701575 5114 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:48:44.703381 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701578 5114 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:48:44.703381 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701581 5114 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:48:44.703381 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701584 5114 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:48:44.703381 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701586 5114 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:48:44.703381 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701589 5114 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:48:44.703381 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701591 5114 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:48:44.703381 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701596 5114 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:48:44.703381 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701599 5114 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:48:44.703381 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701602 5114 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:48:44.703381 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701609 5114 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:48:44.703381 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701612 5114 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:48:44.703381 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701614 5114 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:48:44.703381 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701617 5114 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:48:44.703381 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701620 5114 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:48:44.703381 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701622 5114 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:48:44.703904 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701625 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:48:44.703904 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701628 5114 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:48:44.703904 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701630 5114 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:48:44.703904 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701633 5114 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:48:44.703904 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701635 5114 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:48:44.703904 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701638 5114 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:48:44.703904 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.701643 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:48:44.703904 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702502 5114 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:48:44.703904 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702512 5114 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:48:44.703904 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702516 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:48:44.703904 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702520 5114 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:48:44.703904 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702525 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:48:44.703904 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702529 5114 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:48:44.703904 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702533 5114 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:48:44.703904 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702537 5114 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:48:44.703904 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702541 5114 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:48:44.703904 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702544 5114 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:48:44.703904 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702547 5114 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:48:44.703904 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702549 5114 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:48:44.703904 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702552 5114 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:48:44.704384 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702554 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:48:44.704384 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702556 5114 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:48:44.704384 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702560 5114 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:48:44.704384 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702564 5114 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:48:44.704384 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702567 5114 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:48:44.704384 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702570 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:48:44.704384 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702577 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:48:44.704384 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702580 5114 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:48:44.704384 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702583 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:48:44.704384 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702585 5114 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:48:44.704384 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702588 5114 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:48:44.704384 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702591 5114 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:48:44.704384 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702594 5114 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:48:44.704384 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702596 5114 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:48:44.704384 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702599 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:48:44.704384 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702602 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:48:44.704384 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702604 5114 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:48:44.704384 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702607 5114 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:48:44.704384 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702609 5114 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:48:44.704856 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702613 5114 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:48:44.704856 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702616 5114 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:48:44.704856 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702620 5114 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:48:44.704856 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702622 5114 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:48:44.704856 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702625 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:48:44.704856 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702627 5114 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:48:44.704856 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702630 5114 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:48:44.704856 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702633 5114 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:48:44.704856 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702635 5114 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:48:44.704856 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702637 5114 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:48:44.704856 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702640 5114 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:48:44.704856 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702642 5114 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:48:44.704856 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702645 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:48:44.704856 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702648 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:48:44.704856 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702650 5114 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:48:44.704856 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702653 5114 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:48:44.704856 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702655 5114 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:48:44.704856 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702658 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:48:44.704856 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702660 5114 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:48:44.704856 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702664 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:48:44.705348 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702667 5114 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:48:44.705348 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702670 5114 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:48:44.705348 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702673 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:48:44.705348 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702675 5114 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:48:44.705348 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702678 5114 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:48:44.705348 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702680 5114 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:48:44.705348 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702683 5114 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:48:44.705348 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702686 5114 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:48:44.705348 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702689 5114 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:48:44.705348 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702692 5114 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:48:44.705348 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702703 5114 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:48:44.705348 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702706 5114 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:48:44.705348 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702708 5114 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:48:44.705348 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702718 5114 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:48:44.705348 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702720 5114 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:48:44.705348 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702723 5114 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:48:44.705348 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702727 5114 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:48:44.705348 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702731 5114 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:48:44.705348 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702734 5114 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:48:44.705348 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702737 5114 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:48:44.705864 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702739 5114 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:48:44.705864 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702742 5114 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:48:44.705864 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702744 5114 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:48:44.705864 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702747 5114 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:48:44.705864 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702749 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:48:44.705864 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702752 5114 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:48:44.705864 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702754 5114 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:48:44.705864 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702757 5114 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:48:44.705864 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702759 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:48:44.705864 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702762 5114 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:48:44.705864 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702764 5114 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:48:44.705864 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702769 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:48:44.705864 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702773 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:48:44.705864 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702775 5114 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:48:44.705864 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702778 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:48:44.705864 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702780 5114 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:48:44.705864 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702783 5114 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:48:44.705864 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702785 5114 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:48:44.705864 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702788 5114 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:48:44.706320 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702790 5114 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:48:44.706320 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702792 5114 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:48:44.706320 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702795 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:48:44.706320 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702798 5114 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:48:44.706320 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702800 5114 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:48:44.706320 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702802 5114 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:48:44.706320 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702805 5114 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:48:44.706320 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702815 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:48:44.706320 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702818 5114 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:48:44.706320 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702820 5114 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:48:44.706320 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702823 5114 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:48:44.706320 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702825 5114 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:48:44.706320 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702828 5114 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:48:44.706320 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702830 5114 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:48:44.706320 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.702832 5114 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:48:44.706320 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703416 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:48:44.706320 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703422 5114 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:48:44.706320 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703425 5114 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:48:44.706320 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703428 5114 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:48:44.706801 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703430 5114 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:48:44.706801 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703433 5114 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:48:44.706801 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703438 5114 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:48:44.706801 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703441 5114 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:48:44.706801 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703443 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:48:44.706801 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703446 5114 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:48:44.706801 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703449 5114 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:48:44.706801 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703451 5114 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:48:44.706801 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703454 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:48:44.706801 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703456 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:48:44.706801 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703459 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:48:44.706801 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703462 5114 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:48:44.706801 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703464 5114 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:48:44.706801 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703467 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:48:44.706801 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703470 5114 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:48:44.706801 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703472 5114 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:48:44.706801 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703475 5114 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:48:44.706801 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703492 5114 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:48:44.706801 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703495 5114 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:48:44.706801 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703498 5114 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:48:44.707298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703507 5114 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:48:44.707298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703509 5114 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:48:44.707298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703512 5114 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:48:44.707298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703515 5114 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:48:44.707298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703518 5114 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:48:44.707298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703520 5114 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:48:44.707298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703522 5114 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:48:44.707298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703525 5114 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:48:44.707298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703527 5114 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:48:44.707298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703530 5114 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:48:44.707298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703532 5114 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:48:44.707298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703535 5114 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:48:44.707298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703537 5114 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:48:44.707298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703539 5114 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:48:44.707298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703543 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:48:44.707298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703546 5114 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:48:44.707298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703548 5114 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:48:44.707298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703551 5114 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:48:44.707298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703554 5114 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:48:44.707298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703556 5114 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:48:44.707850 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703559 5114 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:48:44.707850 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703562 5114 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:48:44.707850 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703564 5114 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:48:44.707850 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703566 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:48:44.707850 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703569 5114 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:48:44.707850 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703572 5114 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:48:44.707850 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703574 5114 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:48:44.707850 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703577 5114 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:48:44.707850 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703580 5114 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:48:44.707850 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703582 5114 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:48:44.707850 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703585 5114 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:48:44.707850 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703587 5114 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:48:44.707850 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703590 5114 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:48:44.707850 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703598 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:48:44.707850 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703601 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:48:44.707850 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703603 5114 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:48:44.707850 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703606 5114 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:48:44.707850 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703608 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:48:44.707850 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703611 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:48:44.708323 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703613 5114 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:48:44.708323 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703616 5114 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:48:44.708323 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703618 5114 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:48:44.708323 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703621 5114 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:48:44.708323 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703623 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:48:44.708323 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703626 5114 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:48:44.708323 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703629 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:48:44.708323 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703633 5114 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:48:44.708323 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703637 5114 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:48:44.708323 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703640 5114 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:48:44.708323 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703642 5114 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:48:44.708323 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703645 5114 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:48:44.708323 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703647 5114 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:48:44.708323 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703650 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:48:44.708323 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703652 5114 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:48:44.708323 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703655 5114 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:48:44.708323 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703657 5114 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:48:44.708323 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703660 5114 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:48:44.708323 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703662 5114 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:48:44.709100 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703665 5114 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:48:44.709100 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703668 5114 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:48:44.709100 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703670 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:48:44.709100 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703672 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:48:44.709100 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703675 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:48:44.709100 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703678 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:48:44.709100 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703681 5114 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:48:44.709100 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703683 5114 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:48:44.709100 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703693 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:48:44.709100 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703701 5114 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:48:44.709100 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703704 5114 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:48:44.709100 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703708 5114 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:48:44.709100 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703712 5114 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:48:44.709100 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703715 5114 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:48:44.709100 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703718 5114 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:48:44.709100 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703721 5114 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:48:44.709100 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703723 5114 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:48:44.709100 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703726 5114 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:48:44.709100 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703728 5114 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:48:44.709902 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703731 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:48:44.709902 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703735 5114 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:48:44.709902 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703737 5114 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:48:44.709902 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703740 5114 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:48:44.709902 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.703743 5114 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:48:44.709902 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703862 5114 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 22:48:44.709902 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703878 5114 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 22:48:44.709902 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703885 5114 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 22:48:44.709902 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703890 5114 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 22:48:44.709902 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703894 5114 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 22:48:44.709902 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703898 5114 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 22:48:44.709902 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703902 5114 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 22:48:44.709902 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703906 5114 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 22:48:44.709902 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703910 5114 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 22:48:44.709902 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703913 5114 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 22:48:44.709902 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703917 5114 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 22:48:44.709902 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703921 5114 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 22:48:44.709902 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703924 5114 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 22:48:44.709902 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703927 5114 flags.go:64] FLAG: --cgroup-root="" Nov 02 22:48:44.709902 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703930 5114 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 22:48:44.709902 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703933 5114 flags.go:64] FLAG: --client-ca-file="" Nov 02 22:48:44.709902 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703936 5114 flags.go:64] FLAG: --cloud-provider="external" Nov 02 22:48:44.709902 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703945 5114 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 22:48:44.710493 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703952 5114 flags.go:64] FLAG: --cluster-domain="" Nov 02 22:48:44.710493 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703957 5114 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 22:48:44.710493 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703960 5114 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 22:48:44.710493 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703964 5114 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 22:48:44.710493 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703967 5114 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 22:48:44.710493 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703971 5114 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 22:48:44.710493 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703974 5114 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 22:48:44.710493 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703978 5114 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 22:48:44.710493 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703981 5114 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 22:48:44.710493 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703984 5114 flags.go:64] FLAG: --contention-profiling="false" Nov 02 22:48:44.710493 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703988 5114 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 22:48:44.710493 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703991 5114 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 22:48:44.710493 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703995 5114 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 22:48:44.710493 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.703998 5114 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 22:48:44.710493 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704003 5114 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 22:48:44.710493 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704006 5114 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 22:48:44.710493 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704009 5114 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 22:48:44.710493 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704011 5114 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 22:48:44.710493 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704015 5114 flags.go:64] FLAG: --enable-server="true" Nov 02 22:48:44.710493 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704017 5114 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 22:48:44.710493 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704024 5114 flags.go:64] FLAG: --event-burst="100" Nov 02 22:48:44.710493 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704028 5114 flags.go:64] FLAG: --event-qps="50" Nov 02 22:48:44.710493 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704030 5114 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 22:48:44.710493 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704034 5114 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 22:48:44.711097 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704037 5114 flags.go:64] FLAG: --eviction-hard="" Nov 02 22:48:44.711097 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704042 5114 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 22:48:44.711097 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704045 5114 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 22:48:44.711097 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704048 5114 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 22:48:44.711097 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704051 5114 flags.go:64] FLAG: --eviction-soft="" Nov 02 22:48:44.711097 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704054 5114 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 22:48:44.711097 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704057 5114 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 22:48:44.711097 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704060 5114 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 22:48:44.711097 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704069 5114 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 22:48:44.711097 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704073 5114 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 22:48:44.711097 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704076 5114 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 22:48:44.711097 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704079 5114 flags.go:64] FLAG: --feature-gates="" Nov 02 22:48:44.711097 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704083 5114 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 22:48:44.711097 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704086 5114 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 22:48:44.711097 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704089 5114 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 22:48:44.711097 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704093 5114 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 22:48:44.711097 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704096 5114 flags.go:64] FLAG: --healthz-port="10248" Nov 02 22:48:44.711097 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704099 5114 flags.go:64] FLAG: --help="false" Nov 02 22:48:44.711097 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704104 5114 flags.go:64] FLAG: --hostname-override="ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:48:44.711097 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704108 5114 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 22:48:44.711097 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704111 5114 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 22:48:44.711097 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704114 5114 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 22:48:44.711097 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704117 5114 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 22:48:44.711662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704121 5114 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 22:48:44.711662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704124 5114 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 22:48:44.711662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704127 5114 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 22:48:44.711662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704129 5114 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 22:48:44.711662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704132 5114 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 22:48:44.711662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704135 5114 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 22:48:44.711662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704138 5114 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 22:48:44.711662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704141 5114 flags.go:64] FLAG: --kube-reserved="" Nov 02 22:48:44.711662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704144 5114 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 22:48:44.711662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704147 5114 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 22:48:44.711662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704150 5114 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 22:48:44.711662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704153 5114 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 22:48:44.711662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704156 5114 flags.go:64] FLAG: --lock-file="" Nov 02 22:48:44.711662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704159 5114 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 22:48:44.711662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704162 5114 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 22:48:44.711662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704165 5114 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 22:48:44.711662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704170 5114 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 22:48:44.711662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704173 5114 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 22:48:44.711662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704185 5114 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 22:48:44.711662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704188 5114 flags.go:64] FLAG: --logging-format="text" Nov 02 22:48:44.711662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704191 5114 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 22:48:44.711662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704195 5114 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 22:48:44.711662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704197 5114 flags.go:64] FLAG: --manifest-url="" Nov 02 22:48:44.711662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704200 5114 flags.go:64] FLAG: --manifest-url-header="" Nov 02 22:48:44.711662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704205 5114 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704208 5114 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704212 5114 flags.go:64] FLAG: --max-pods="110" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704216 5114 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704220 5114 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704223 5114 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704226 5114 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704229 5114 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704232 5114 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704234 5114 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704242 5114 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704245 5114 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704248 5114 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704251 5114 flags.go:64] FLAG: --pod-cidr="" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704254 5114 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704257 5114 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704260 5114 flags.go:64] FLAG: --pods-per-core="0" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704263 5114 flags.go:64] FLAG: --port="10250" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704266 5114 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704269 5114 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0900d3cfc04b8d8b0" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704272 5114 flags.go:64] FLAG: --qos-reserved="" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704275 5114 flags.go:64] FLAG: --read-only-port="10255" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704278 5114 flags.go:64] FLAG: --register-node="true" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704281 5114 flags.go:64] FLAG: --register-with-taints="" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704284 5114 flags.go:64] FLAG: --registry-burst="10" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704288 5114 flags.go:64] FLAG: --registry-qps="5" Nov 02 22:48:44.712257 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704292 5114 flags.go:64] FLAG: --reserved-cpus="" Nov 02 22:48:44.712985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704301 5114 flags.go:64] FLAG: --reserved-memory="" Nov 02 22:48:44.712985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704305 5114 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 22:48:44.712985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704308 5114 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 22:48:44.712985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704311 5114 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 22:48:44.712985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704314 5114 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 22:48:44.712985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704317 5114 flags.go:64] FLAG: --runonce="false" Nov 02 22:48:44.712985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704320 5114 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 22:48:44.712985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704323 5114 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 22:48:44.712985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704328 5114 flags.go:64] FLAG: --seccomp-default="false" Nov 02 22:48:44.712985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704331 5114 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 22:48:44.712985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704334 5114 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 22:48:44.712985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704337 5114 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 22:48:44.712985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704340 5114 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 22:48:44.712985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704343 5114 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 22:48:44.712985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704346 5114 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 22:48:44.712985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704350 5114 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 22:48:44.712985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704353 5114 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 22:48:44.712985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704356 5114 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 22:48:44.712985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704359 5114 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 22:48:44.712985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704362 5114 flags.go:64] FLAG: --system-cgroups="" Nov 02 22:48:44.712985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704365 5114 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 22:48:44.712985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704370 5114 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 22:48:44.712985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704373 5114 flags.go:64] FLAG: --tls-cert-file="" Nov 02 22:48:44.712985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704376 5114 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 22:48:44.712985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704383 5114 flags.go:64] FLAG: --tls-min-version="" Nov 02 22:48:44.713607 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704386 5114 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 22:48:44.713607 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704389 5114 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 22:48:44.713607 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704392 5114 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 22:48:44.713607 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704395 5114 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 22:48:44.713607 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704398 5114 flags.go:64] FLAG: --v="2" Nov 02 22:48:44.713607 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704403 5114 flags.go:64] FLAG: --version="false" Nov 02 22:48:44.713607 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704409 5114 flags.go:64] FLAG: --vmodule="" Nov 02 22:48:44.713607 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704413 5114 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 22:48:44.713607 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704422 5114 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 22:48:44.713607 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704627 5114 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:48:44.713607 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704633 5114 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:48:44.713607 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704636 5114 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:48:44.713607 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704639 5114 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:48:44.713607 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704642 5114 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:48:44.713607 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704645 5114 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:48:44.713607 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704650 5114 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:48:44.713607 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704652 5114 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:48:44.713607 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704655 5114 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:48:44.713607 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704658 5114 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:48:44.713607 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704661 5114 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:48:44.713607 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704663 5114 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:48:44.713607 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704666 5114 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:48:44.714149 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704668 5114 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:48:44.714149 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704671 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:48:44.714149 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704674 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:48:44.714149 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704676 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:48:44.714149 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704679 5114 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:48:44.714149 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704681 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:48:44.714149 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704685 5114 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:48:44.714149 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704690 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:48:44.714149 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704694 5114 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:48:44.714149 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704697 5114 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:48:44.714149 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704700 5114 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:48:44.714149 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704702 5114 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:48:44.714149 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704705 5114 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:48:44.714149 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704707 5114 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:48:44.714149 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704710 5114 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:48:44.714149 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704712 5114 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:48:44.714149 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704717 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:48:44.714149 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704719 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:48:44.714149 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704721 5114 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:48:44.714646 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704730 5114 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:48:44.714646 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704733 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:48:44.714646 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704735 5114 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:48:44.714646 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704738 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:48:44.714646 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704740 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:48:44.714646 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704743 5114 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:48:44.714646 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704747 5114 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:48:44.714646 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704750 5114 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:48:44.714646 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704754 5114 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:48:44.714646 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704757 5114 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:48:44.714646 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704760 5114 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:48:44.714646 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704764 5114 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:48:44.714646 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704766 5114 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:48:44.714646 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704769 5114 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:48:44.714646 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704772 5114 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:48:44.714646 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704774 5114 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:48:44.714646 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704777 5114 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:48:44.714646 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704779 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:48:44.714646 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704782 5114 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:48:44.714646 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704784 5114 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:48:44.715136 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704787 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:48:44.715136 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704790 5114 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:48:44.715136 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704792 5114 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:48:44.715136 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704794 5114 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:48:44.715136 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704797 5114 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:48:44.715136 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704799 5114 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:48:44.715136 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704802 5114 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:48:44.715136 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704804 5114 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:48:44.715136 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704807 5114 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:48:44.715136 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704810 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:48:44.715136 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704813 5114 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:48:44.715136 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704815 5114 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:48:44.715136 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704818 5114 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:48:44.715136 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704827 5114 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:48:44.715136 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704830 5114 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:48:44.715136 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704832 5114 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:48:44.715136 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704835 5114 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:48:44.715136 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704837 5114 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:48:44.715136 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704842 5114 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:48:44.715136 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704844 5114 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:48:44.715675 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704847 5114 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:48:44.715675 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704849 5114 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:48:44.715675 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704852 5114 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:48:44.715675 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704854 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:48:44.715675 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704857 5114 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:48:44.715675 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704859 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:48:44.715675 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704862 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:48:44.715675 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704865 5114 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:48:44.715675 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704867 5114 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:48:44.715675 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704870 5114 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:48:44.715675 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704872 5114 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:48:44.715675 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704875 5114 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:48:44.715675 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704878 5114 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:48:44.715675 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704880 5114 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:48:44.715675 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704883 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:48:44.715675 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704885 5114 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:48:44.715675 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704887 5114 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:48:44.715675 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704890 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:48:44.715675 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704893 5114 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:48:44.715675 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704895 5114 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:48:44.716402 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704897 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:48:44.716402 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704901 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:48:44.716402 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704904 5114 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:48:44.716402 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704906 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:48:44.716402 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704909 5114 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:48:44.716402 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704911 5114 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:48:44.716402 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704919 5114 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:48:44.716402 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704922 5114 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:48:44.716402 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704924 5114 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:48:44.716402 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704926 5114 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:48:44.716402 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704930 5114 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:48:44.716402 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704933 5114 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:48:44.716402 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704935 5114 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:48:44.716402 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.704941 5114 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:48:44.716402 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.704982 5114 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:48:44.716402 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705027 5114 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:48:44.717080 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705032 5114 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:48:44.717080 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705036 5114 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:48:44.717080 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705040 5114 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:48:44.717080 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705043 5114 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:48:44.717080 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705046 5114 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:48:44.717080 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705048 5114 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:48:44.717080 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705051 5114 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:48:44.717080 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705053 5114 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:48:44.717080 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705056 5114 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:48:44.717080 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705058 5114 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:48:44.717080 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705061 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:48:44.717080 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705063 5114 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:48:44.717080 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705066 5114 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:48:44.717080 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705069 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:48:44.717080 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705071 5114 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:48:44.717080 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705074 5114 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:48:44.717080 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705076 5114 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:48:44.717080 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705084 5114 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:48:44.717080 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705086 5114 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:48:44.717080 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705089 5114 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:48:44.717834 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705091 5114 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:48:44.717834 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705094 5114 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:48:44.717834 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705096 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:48:44.717834 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705105 5114 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:48:44.717834 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705108 5114 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:48:44.717834 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705116 5114 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:48:44.717834 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705118 5114 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:48:44.717834 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705124 5114 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:48:44.717834 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705127 5114 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:48:44.717834 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705129 5114 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:48:44.717834 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705132 5114 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:48:44.717834 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705134 5114 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:48:44.717834 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705137 5114 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:48:44.717834 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705139 5114 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:48:44.717834 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705142 5114 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:48:44.717834 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705144 5114 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:48:44.717834 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705147 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:48:44.717834 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705149 5114 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:48:44.717834 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705152 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:48:44.717834 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705154 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:48:44.718549 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705157 5114 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:48:44.718549 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705159 5114 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:48:44.718549 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705162 5114 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:48:44.718549 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705164 5114 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:48:44.718549 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705167 5114 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:48:44.718549 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705169 5114 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:48:44.718549 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705171 5114 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:48:44.718549 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705174 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:48:44.718549 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705176 5114 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:48:44.718549 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705180 5114 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:48:44.718549 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705182 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:48:44.718549 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705185 5114 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:48:44.718549 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705187 5114 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:48:44.718549 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705190 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:48:44.718549 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705192 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:48:44.718549 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705195 5114 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:48:44.718549 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705203 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:48:44.718549 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705206 5114 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:48:44.718549 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705208 5114 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:48:44.718549 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705213 5114 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:48:44.719046 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705215 5114 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:48:44.719046 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705218 5114 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:48:44.719046 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705220 5114 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:48:44.719046 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705223 5114 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:48:44.719046 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705225 5114 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:48:44.719046 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705228 5114 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:48:44.719046 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705230 5114 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:48:44.719046 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705233 5114 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:48:44.719046 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705235 5114 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:48:44.719046 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705238 5114 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:48:44.719046 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705240 5114 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:48:44.719046 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705243 5114 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:48:44.719046 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705246 5114 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:48:44.719046 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705248 5114 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:48:44.719046 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705251 5114 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:48:44.719046 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705253 5114 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:48:44.719046 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705256 5114 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:48:44.719046 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705258 5114 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:48:44.719046 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705260 5114 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:48:44.719046 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705263 5114 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:48:44.719604 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705265 5114 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:48:44.719604 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705269 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:48:44.719604 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705271 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:48:44.719604 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705274 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:48:44.719604 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705276 5114 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:48:44.719604 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705279 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:48:44.719604 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705291 5114 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:48:44.719604 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705295 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:48:44.719604 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705298 5114 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:48:44.719604 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705306 5114 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:48:44.719604 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705309 5114 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:48:44.719604 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705313 5114 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:48:44.719604 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705315 5114 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:48:44.719604 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705318 5114 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:48:44.719604 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705320 5114 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:48:44.719604 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705323 5114 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:48:44.719604 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705326 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:48:44.719604 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705328 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:48:44.719604 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705340 5114 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:48:44.720298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705343 5114 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:48:44.720298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705345 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:48:44.720298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705348 5114 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:48:44.720298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705350 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:48:44.720298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705354 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:48:44.720298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.705357 5114 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:48:44.720298 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.711457 5114 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 22:48:44.720298 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.711499 5114 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 22:48:44.720298 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.711525 5114 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 22:48:44.720298 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.711531 5114 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 22:48:44.720298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711610 5114 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:48:44.720298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711616 5114 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:48:44.720298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711619 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:48:44.720298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711621 5114 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:48:44.720298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711624 5114 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:48:44.720298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711627 5114 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:48:44.720298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711630 5114 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:48:44.720298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711632 5114 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:48:44.720298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711635 5114 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:48:44.720298 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711638 5114 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:48:44.720972 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711641 5114 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:48:44.720972 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711644 5114 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:48:44.720972 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711646 5114 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:48:44.720972 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711649 5114 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:48:44.720972 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711652 5114 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:48:44.720972 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711654 5114 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:48:44.720972 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711657 5114 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:48:44.720972 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711659 5114 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:48:44.720972 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711662 5114 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:48:44.720972 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711665 5114 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:48:44.720972 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711668 5114 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:48:44.720972 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711670 5114 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:48:44.720972 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711673 5114 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:48:44.720972 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711676 5114 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:48:44.720972 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711679 5114 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:48:44.720972 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711681 5114 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:48:44.720972 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711684 5114 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:48:44.720972 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711686 5114 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:48:44.720972 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711690 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:48:44.720972 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711693 5114 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:48:44.721476 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711695 5114 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:48:44.721476 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711698 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:48:44.721476 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711701 5114 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:48:44.721476 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711704 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:48:44.721476 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711706 5114 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:48:44.721476 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711709 5114 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:48:44.721476 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711711 5114 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:48:44.721476 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711714 5114 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:48:44.721476 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711716 5114 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:48:44.721476 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711719 5114 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:48:44.721476 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711721 5114 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:48:44.721476 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711724 5114 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:48:44.721476 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711727 5114 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:48:44.721476 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711729 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:48:44.721476 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711732 5114 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:48:44.721476 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711734 5114 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:48:44.721476 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711743 5114 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:48:44.721476 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711745 5114 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:48:44.721476 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711749 5114 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:48:44.721476 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711752 5114 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:48:44.721980 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711754 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:48:44.721980 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711757 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:48:44.721980 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711759 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:48:44.721980 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711762 5114 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:48:44.721980 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711764 5114 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:48:44.721980 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711767 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:48:44.721980 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711770 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:48:44.721980 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711772 5114 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:48:44.721980 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711776 5114 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:48:44.721980 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711781 5114 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:48:44.721980 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711785 5114 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:48:44.721980 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711790 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:48:44.721980 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711794 5114 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:48:44.721980 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711799 5114 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:48:44.721980 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711803 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:48:44.721980 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711807 5114 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:48:44.721980 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711811 5114 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:48:44.721980 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711815 5114 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:48:44.721980 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711818 5114 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:48:44.722438 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711821 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:48:44.722438 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711823 5114 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:48:44.722438 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711826 5114 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:48:44.722438 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711828 5114 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:48:44.722438 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711831 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:48:44.722438 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711834 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:48:44.722438 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711838 5114 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:48:44.722438 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711842 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:48:44.722438 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711845 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:48:44.722438 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711849 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:48:44.722438 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711853 5114 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:48:44.722438 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711856 5114 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:48:44.722438 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711859 5114 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:48:44.722438 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711861 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:48:44.722438 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711864 5114 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:48:44.722438 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711867 5114 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:48:44.722438 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711869 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:48:44.722438 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711872 5114 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:48:44.722438 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711874 5114 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:48:44.722937 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711877 5114 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:48:44.722937 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711879 5114 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:48:44.722937 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711883 5114 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:48:44.722937 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711887 5114 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:48:44.722937 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711890 5114 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:48:44.722937 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711894 5114 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:48:44.722937 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711896 5114 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:48:44.722937 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711899 5114 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:48:44.722937 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711901 5114 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:48:44.722937 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711904 5114 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:48:44.722937 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711906 5114 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:48:44.722937 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711909 5114 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:48:44.722937 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711912 5114 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:48:44.722937 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711914 5114 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:48:44.722937 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711917 5114 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:48:44.722937 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711919 5114 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:48:44.722937 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711922 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:48:44.722937 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.711924 5114 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:48:44.723404 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.711957 5114 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:48:44.723404 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712113 5114 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:48:44.723404 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712118 5114 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:48:44.723404 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712121 5114 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:48:44.723404 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712124 5114 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:48:44.723404 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712128 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:48:44.723404 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712131 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:48:44.723404 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712135 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:48:44.723404 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712137 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:48:44.723404 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712140 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:48:44.723404 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712142 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:48:44.723404 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712145 5114 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:48:44.723404 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712148 5114 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:48:44.723404 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712150 5114 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:48:44.723404 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712153 5114 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:48:44.723404 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712156 5114 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:48:44.723858 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712158 5114 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:48:44.723858 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712161 5114 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:48:44.723858 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712163 5114 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:48:44.723858 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712166 5114 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:48:44.723858 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712168 5114 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:48:44.723858 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712171 5114 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:48:44.723858 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712174 5114 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:48:44.723858 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712177 5114 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:48:44.723858 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712179 5114 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:48:44.723858 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712182 5114 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:48:44.723858 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712184 5114 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:48:44.723858 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712187 5114 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:48:44.723858 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712189 5114 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:48:44.723858 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712193 5114 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:48:44.723858 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712197 5114 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:48:44.723858 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712200 5114 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:48:44.723858 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712203 5114 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:48:44.723858 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712206 5114 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:48:44.723858 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712209 5114 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:48:44.723858 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712211 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:48:44.724342 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712214 5114 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:48:44.724342 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712217 5114 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:48:44.724342 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712219 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:48:44.724342 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712223 5114 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:48:44.724342 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712226 5114 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:48:44.724342 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712228 5114 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:48:44.724342 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712231 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:48:44.724342 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712233 5114 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:48:44.724342 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712236 5114 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:48:44.724342 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712238 5114 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:48:44.724342 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712240 5114 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:48:44.724342 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712243 5114 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:48:44.724342 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712247 5114 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:48:44.724342 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712251 5114 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:48:44.724342 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712253 5114 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:48:44.724342 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712256 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:48:44.724342 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712259 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:48:44.724342 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712262 5114 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:48:44.724342 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712265 5114 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:48:44.724952 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712267 5114 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:48:44.724952 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712270 5114 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:48:44.724952 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712273 5114 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:48:44.724952 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712275 5114 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:48:44.724952 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712278 5114 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:48:44.724952 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712280 5114 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:48:44.724952 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712283 5114 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:48:44.724952 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712286 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:48:44.724952 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712288 5114 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:48:44.724952 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712290 5114 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:48:44.724952 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712293 5114 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:48:44.724952 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712296 5114 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:48:44.724952 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712298 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:48:44.724952 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712301 5114 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:48:44.724952 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712303 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:48:44.724952 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712305 5114 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:48:44.724952 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712309 5114 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:48:44.724952 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712312 5114 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:48:44.724952 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712315 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:48:44.725472 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712317 5114 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:48:44.725472 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712320 5114 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:48:44.725472 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712322 5114 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:48:44.725472 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712325 5114 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:48:44.725472 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712327 5114 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:48:44.725472 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712329 5114 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:48:44.725472 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712332 5114 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:48:44.725472 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712334 5114 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:48:44.725472 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712337 5114 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:48:44.725472 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712340 5114 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:48:44.725472 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712342 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:48:44.725472 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712345 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:48:44.725472 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712347 5114 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:48:44.725472 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712350 5114 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:48:44.725472 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712353 5114 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:48:44.725472 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712355 5114 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:48:44.725472 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712358 5114 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:48:44.725472 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712360 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:48:44.725472 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712362 5114 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:48:44.725472 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712365 5114 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:48:44.726148 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712367 5114 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:48:44.726148 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712369 5114 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:48:44.726148 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712372 5114 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:48:44.726148 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712374 5114 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:48:44.726148 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712376 5114 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:48:44.726148 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712379 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:48:44.726148 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712381 5114 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:48:44.726148 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712384 5114 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:48:44.726148 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712386 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:48:44.726148 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712389 5114 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:48:44.726148 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712392 5114 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:48:44.726148 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712395 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:48:44.726148 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712398 5114 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:48:44.726148 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.712421 5114 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:48:44.726148 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712453 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:48:44.726148 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712457 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:48:44.726818 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712460 5114 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:48:44.726818 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712463 5114 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:48:44.726818 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712465 5114 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:48:44.726818 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712468 5114 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:48:44.726818 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712471 5114 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:48:44.726818 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712474 5114 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:48:44.726818 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712476 5114 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:48:44.726818 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712492 5114 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:48:44.726818 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712495 5114 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:48:44.726818 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712498 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:48:44.726818 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712501 5114 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:48:44.726818 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712504 5114 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:48:44.726818 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712507 5114 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:48:44.726818 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712510 5114 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:48:44.726818 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712512 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:48:44.726818 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712515 5114 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:48:44.726818 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712517 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:48:44.726818 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712520 5114 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:48:44.726818 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712522 5114 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:48:44.727679 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712525 5114 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:48:44.727679 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712527 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:48:44.727679 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712530 5114 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:48:44.727679 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712532 5114 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:48:44.727679 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712534 5114 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:48:44.727679 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712537 5114 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:48:44.727679 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712539 5114 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:48:44.727679 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712542 5114 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:48:44.727679 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712545 5114 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:48:44.727679 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712548 5114 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:48:44.727679 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712551 5114 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:48:44.727679 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712554 5114 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:48:44.727679 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712556 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:48:44.727679 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712559 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:48:44.727679 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712561 5114 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:48:44.727679 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712564 5114 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:48:44.727679 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712567 5114 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:48:44.727679 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712569 5114 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:48:44.727679 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712572 5114 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:48:44.727679 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712575 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:48:44.728330 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712577 5114 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:48:44.728330 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712580 5114 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:48:44.728330 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712582 5114 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:48:44.728330 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712585 5114 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:48:44.728330 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712587 5114 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:48:44.728330 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712590 5114 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:48:44.728330 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712592 5114 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:48:44.728330 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712595 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:48:44.728330 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712597 5114 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:48:44.728330 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712600 5114 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:48:44.728330 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712602 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:48:44.728330 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712605 5114 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:48:44.728330 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712607 5114 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:48:44.728330 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712610 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:48:44.728330 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712612 5114 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:48:44.728330 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712614 5114 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:48:44.728330 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712617 5114 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:48:44.728330 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712620 5114 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:48:44.728330 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712622 5114 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:48:44.728330 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712625 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:48:44.729076 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712627 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:48:44.729076 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712630 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:48:44.729076 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712633 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:48:44.729076 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712636 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:48:44.729076 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712638 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:48:44.729076 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712640 5114 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:48:44.729076 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712643 5114 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:48:44.729076 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712647 5114 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:48:44.729076 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712649 5114 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:48:44.729076 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712652 5114 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:48:44.729076 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712654 5114 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:48:44.729076 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712656 5114 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:48:44.729076 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712659 5114 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:48:44.729076 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712661 5114 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:48:44.729076 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712664 5114 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:48:44.729076 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712666 5114 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:48:44.729076 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712669 5114 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:48:44.729076 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712672 5114 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:48:44.729076 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712675 5114 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:48:44.729076 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712677 5114 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:48:44.734796 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712680 5114 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:48:44.734796 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712682 5114 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:48:44.734796 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712685 5114 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:48:44.734796 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712688 5114 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:48:44.734796 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712692 5114 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:48:44.734796 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712695 5114 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:48:44.734796 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712697 5114 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:48:44.734796 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712700 5114 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:48:44.734796 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712703 5114 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:48:44.734796 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712705 5114 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:48:44.734796 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712711 5114 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:48:44.734796 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712714 5114 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:48:44.734796 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712716 5114 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:48:44.734796 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712719 5114 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:48:44.734796 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712721 5114 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:48:44.734796 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712724 5114 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:48:44.734796 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712727 5114 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:48:44.734796 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712729 5114 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:48:44.734796 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712732 5114 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:48:44.734796 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712734 5114 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:48:44.735304 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712737 5114 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:48:44.735304 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712740 5114 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:48:44.735304 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712745 5114 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:48:44.735304 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712749 5114 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:48:44.735304 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:48:44.712752 5114 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:48:44.735304 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.712824 5114 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 22:48:44.735304 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.713892 5114 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 22:48:44.735304 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.713959 5114 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 22:48:44.735304 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.714156 5114 server.go:1014] "Starting client certificate rotation" Nov 02 22:48:44.735304 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.714299 5114 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 22:48:44.735304 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.714353 5114 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 22:42:33 +0000 UTC" deadline="2028-06-03 07:13:54.002543206 +0000 UTC" Nov 02 22:48:44.735304 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.714391 5114 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13880h25m9.288153974s" Nov 02 22:48:44.735304 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.714668 5114 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:48:44.735304 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.715874 5114 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:48:44.735304 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.717864 5114 log.go:25] "Validated CRI v1 runtime API" Nov 02 22:48:44.735304 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.719635 5114 log.go:25] "Validated CRI v1 image API" Nov 02 22:48:44.735758 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.720435 5114 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 22:48:44.735758 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.722447 5114 fs.go:135] Filesystem UUIDs: map[673ecb43-8824-46ab-b3a9-31f332ba0903:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 86fde7a3-20a9-464e-bd82-e9a95e267ec3:/dev/nvme0n1p4] Nov 02 22:48:44.735828 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.722468 5114 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00d4ccbee98a6deedb9d504bf767e5a7a9cf38c0c1f85f8ba00911ad35ff1ab6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00d4ccbee98a6deedb9d504bf767e5a7a9cf38c0c1f85f8ba00911ad35ff1ab6/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1891a2fee619b7749bd605bce66982e33fdf968cd5798254a682a85b5afe31a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1891a2fee619b7749bd605bce66982e33fdf968cd5798254a682a85b5afe31a2/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c8a3a269fc77b0500375f98669c75aa984112b8933bf283336a1eafd429f027/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c8a3a269fc77b0500375f98669c75aa984112b8933bf283336a1eafd429f027/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d746a2fbc6bc62606a9cdbe5304b36eed622f947615dd7572e7fd331e4ef8cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d746a2fbc6bc62606a9cdbe5304b36eed622f947615dd7572e7fd331e4ef8cb/userdata/shm major:0 minor:299 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31f0641478a464484c9f4d8b85a2165de817c41bc124000b33749a28401bba05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31f0641478a464484c9f4d8b85a2165de817c41bc124000b33749a28401bba05/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/39117eeba9a2f4d86737c81b31806feef362b7e62df1d7fb6acf06b9904ba182/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/39117eeba9a2f4d86737c81b31806feef362b7e62df1d7fb6acf06b9904ba182/userdata/shm major:0 minor:303 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3eff20b4dd4839ad801ecc8d1fd85f714de2dc60127bd547b37c0912bfafb0ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3eff20b4dd4839ad801ecc8d1fd85f714de2dc60127bd547b37c0912bfafb0ef/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4468bd03a890b8fae2c6d143db0a84a5e10601fc853fd2ae6e9c4c61b5481300/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4468bd03a890b8fae2c6d143db0a84a5e10601fc853fd2ae6e9c4c61b5481300/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/50a8014b3378c3ec8a909389a05a913ec59d4a9b30c4a80f1efd93bdceceaf01/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/50a8014b3378c3ec8a909389a05a913ec59d4a9b30c4a80f1efd93bdceceaf01/userdata/shm major:0 minor:295 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d00480a202d2e0f59c4e97f43655a9ebc9742638459c8d5eac352dc1c64aee5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d00480a202d2e0f59c4e97f43655a9ebc9742638459c8d5eac352dc1c64aee5/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/754fedf886b43368b928ff4a1d861296eb55a85d000114da6a782f4f41fbdf89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/754fedf886b43368b928ff4a1d861296eb55a85d000114da6a782f4f41fbdf89/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8fd672dc0cc08aad560279c4603970941fb1575285bcbea70018da387a77c42a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8fd672dc0cc08aad560279c4603970941fb1575285bcbea70018da387a77c42a/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/901a3af424db738e9f6f250dd427c12d5a46087d7b5623159f66f7c229fd230f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/901a3af424db738e9f6f250dd427c12d5a46087d7b5623159f66f7c229fd230f/userdata/shm major:0 minor:337 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90ca4505d28c37a4e751200f1e0076d676713d8b2963645e83c2e23b434563e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90ca4505d28c37a4e751200f1e0076d676713d8b2963645e83c2e23b434563e6/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9887e0ad9b044eb9cca02058d29958f530800fbc5768f2445a8c7f229b674f76/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9887e0ad9b044eb9cca02058d29958f530800fbc5768f2445a8c7f229b674f76/userdata/shm major:0 minor:197 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a6201c52a6605c5c878d4d4889bec6411be08be8372537ce6b6e5fc1c29f333/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a6201c52a6605c5c878d4d4889bec6411be08be8372537ce6b6e5fc1c29f333/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e95567ad1efeb49460dea7e0e951b3c0c82b7739e5d86b3220a03cb0f25a531/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e95567ad1efeb49460dea7e0e951b3c0c82b7739e5d86b3220a03cb0f25a531/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af5ad537206c6aee77ca429dbccb84de80c1cd0c900448f27ec20ef813a5f69d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af5ad537206c6aee77ca429dbccb84de80c1cd0c900448f27ec20ef813a5f69d/userdata/shm major:0 minor:339 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b113915b35fee03e088cb22493cd9621f8a66cce5eb9036eb4ccde329bbf16ff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b113915b35fee03e088cb22493cd9621f8a66cce5eb9036eb4ccde329bbf16ff/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b77d42eb9ea696f91f01f98fa13c0571e8ecbc3d8d5e0039236a7fe32d46c502/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b77d42eb9ea696f91f01f98fa13c0571e8ecbc3d8d5e0039236a7fe32d46c502/userdata/shm major:0 minor:277 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e93ae72ab7af8d8f988c835df9607896d50473144bc74bfb5a4d8038e771102d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e93ae72ab7af8d8f988c835df9607896d50473144bc74bfb5a4d8038e771102d/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc5798118300a384cf4aba07c07566f07b52867f51baaf40da43e1569a39264c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc5798118300a384cf4aba07c07566f07b52867f51baaf40da43e1569a39264c/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a5e437f-4589-4f76-b777-026186f06c73/volumes/kubernetes.io~projected/kube-api-access-66b2k:{mountpoint:/var/lib/kubelet/pods/0a5e437f-4589-4f76-b777-026186f06c73/volumes/kubernetes.io~projected/kube-api-access-66b2k major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c414822-2ac4-4cd7-844f-a1f334dba25a/volumes/kubernetes.io~projected/kube-api-access-gw64f:{mountpoint:/var/lib/kubelet/pods/0c414822-2ac4-4cd7-844f-a1f334dba25a/volumes/kubernetes.io~projected/kube-api-access-gw64f major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c414822-2ac4-4cd7-844f-a1f334dba25a/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0c414822-2ac4-4cd7-844f-a1f334dba25a/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c414822-2ac4-4cd7-844f-a1f334dba25a/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/0c414822-2ac4-4cd7-844f-a1f334dba25a/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e82120d-ed55-489b-a3d5-a553ac1b1678/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0e82120d-ed55-489b-a3d5-a553ac1b1678/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e82120d-ed55-489b-a3d5-a553ac1b1678/volumes/kubernetes.io~projected/kube-api-access-sj75b:{mountpoint:/var/lib/kubelet/pods/0e82120d-ed55-489b-a3d5-a553ac1b1678/volumes/kubernetes.io~projected/kube-api-access-sj75b major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e82120d-ed55-489b-a3d5-a553ac1b1678/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0e82120d-ed55-489b-a3d5-a553ac1b1678/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1062888a-e10c-4a34-b84a-f85ff9eb8063/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/1062888a-e10c-4a34-b84a-f85ff9eb8063/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1322aa24-24c5-4a2e-b767-9bda95b4daaa/volumes/kubernetes.io~projected/kube-api-access-rl98f:{mountpoint:/var/lib/kubelet/pods/1322aa24-24c5-4a2e-b767-9bda95b4daaa/volumes/kubernetes.io~projected/kube-api-access-rl98f major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1322aa24-24c5-4a2e-b767-9bda95b4daaa/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/1322aa24-24c5-4a2e-b767-9bda95b4daaa/volumes/kubernetes.io~secret/cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32856eec-17a2-4d3a-8ca1-53755398f81c/volumes/kubernetes.io~projected/kube-api-access-pg6gf:{mountpoint:/var/lib/kubelet/pods/32856eec-17a2-4d3a-8ca1-53755398f81c/volumes/kubernetes.io~projected/kube-api-access-pg6gf major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38ed00fd-8810-4f2d-868e-6ab2edcb860e/volumes/kubernetes.io~projected/kube-api-access-t6655:{mountpoint:/var/lib/kubelet/pods/38ed00fd-8810-4f2d-868e-6ab2edcb860e/volumes/kubernetes.io~projected/kube-api-access-t6655 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38ed00fd-8810-4f2d-868e-6ab2edcb860e/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/38ed00fd-8810-4f2d-868e-6ab2edcb860e/volumes/kubernetes.io~secret/metrics-tls major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ce08d97-9151-4a82-92b3-a01a4057f046/volumes/kubernetes.io~projected/kube-api-access-qpfqn:{mountpoint:/var/lib/kubelet/pods/5ce08d97-9151-4a82-92b3-a01a4057f046/volumes/kubernetes.io~projected/kube-api-access-qpfqn major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ce08d97-9151-4a82-92b3-a01a4057f046/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/5ce08d97-9151-4a82-92b3-a01a4057f046/volumes/kubernetes.io~secret/metrics-certs major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73682abc-fb01-4ac2-a073-fcffad51a512/volumes/kubernetes.io~projected/kube-api-access-nnvrv:{mountpoint:/var/lib/kubelet/pods/73682abc-fb01-4ac2-a073-fcffad51a512/volumes/kubernetes.io~projected/kube-api-access-nnvrv major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94071069-300b-43de-87e3-58e31ac8ab9d/volumes/kubernetes.io~projected/kube-api-access-pw7mm:{mountpoint:/var/lib/kubelet/pods/94071069-300b-43de-87e3-58e31ac8ab9d/volumes/kubernetes.io~projected/kube-api-access-pw7mm major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94071069-300b-43de-87e3-58e31ac8ab9d/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/94071069-300b-43de-87e3-58e31ac8ab9d/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94071069-300b-43de-87e3-58e31ac8ab9d/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/94071069-300b-43de-87e3-58e31ac8ab9d/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad69acdc-e65a-4aad-ac33-354f6ea07401/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/ad69acdc-e65a-4aad-ac33-354f6ea07401/volumes/kubernetes.io~secret/tls-certificates major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6048231-1f5f-482a-af32-228a7c75f910/volumes/kubernetes.io~projected/kube-api-access-m7zxb:{mountpoint:/var/lib/kubelet/pods/b6048231-1f5f-482a-af32-228a7c75f910/volumes/kubernetes.io~projected/kube-api-access-m7zxb major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d/volumes/kubernetes.io~projected/kube-api-access-77psj:{mountpoint:/var/lib/kubelet/pods/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d/volumes/kubernetes.io~projected/kube-api-access-77psj major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca56f427-3734-48ec-b7ce-100daeb7b3bb/volumes/kubernetes.io~projected/kube-api-access-pcgff:{mountpoint:/var/lib/kubelet/pods/ca56f427-3734-48ec-b7ce-100daeb7b3bb/volumes/kubernetes.io~projected/kube-api-access-pcgff major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0121629-a394-4f12-a90c-2f4246436bc5/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/d0121629-a394-4f12-a90c-2f4246436bc5/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0121629-a394-4f12-a90c-2f4246436bc5/volumes/kubernetes.io~projected/kube-api-access-5mh6p:{mountpoint:/var/lib/kubelet/pods/d0121629-a394-4f12-a90c-2f4246436bc5/volumes/kubernetes.io~projected/kube-api-access-5mh6p major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0121629-a394-4f12-a90c-2f4246436bc5/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/d0121629-a394-4f12-a90c-2f4246436bc5/volumes/kubernetes.io~projected/registry-tls major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0121629-a394-4f12-a90c-2f4246436bc5/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/d0121629-a394-4f12-a90c-2f4246436bc5/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0121629-a394-4f12-a90c-2f4246436bc5/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/d0121629-a394-4f12-a90c-2f4246436bc5/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d05df5bf-8c7c-4c79-afa6-96fc7d3689e4/volumes/kubernetes.io~projected/kube-api-access-cm5cb:{mountpoint:/var/lib/kubelet/pods/d05df5bf-8c7c-4c79-afa6-96fc7d3689e4/volumes/kubernetes.io~projected/kube-api-access-cm5cb major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db5f685d-3887-4e83-99f8-7538f184224f/volumes/kubernetes.io~projected/kube-api-access-tr6fc:{mountpoint:/var/lib/kubelet/pods/db5f685d-3887-4e83-99f8-7538f184224f/volumes/kubernetes.io~projected/kube-api-access-tr6fc major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db5f685d-3887-4e83-99f8-7538f184224f/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/db5f685d-3887-4e83-99f8-7538f184224f/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e608a3bc-a7f2-45a1-9603-d85bbbe20614/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/e608a3bc-a7f2-45a1-9603-d85bbbe20614/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee507e21-156a-48a5-9c73-5f9e98bdd44a/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/ee507e21-156a-48a5-9c73-5f9e98bdd44a/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee507e21-156a-48a5-9c73-5f9e98bdd44a/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/ee507e21-156a-48a5-9c73-5f9e98bdd44a/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee507e21-156a-48a5-9c73-5f9e98bdd44a/volumes/kubernetes.io~projected/kube-api-access-6j2w5:{mountpoint:/var/lib/kubelet/pods/ee507e21-156a-48a5-9c73-5f9e98bdd44a/volumes/kubernetes.io~projected/kube-api-access-6j2w5 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4/volumes/kubernetes.io~projected/kube-api-access-kxxkn:{mountpoint:/var/lib/kubelet/pods/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4/volumes/kubernetes.io~projected/kube-api-access-kxxkn major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fabb0266-226c-4935-a9fc-5ed6199e2f9b/volumes/kubernetes.io~projected/kube-api-access-9lppg:{mountpoint:/var/lib/kubelet/pods/fabb0266-226c-4935-a9fc-5ed6199e2f9b/volumes/kubernetes.io~projected/kube-api-access-9lppg major:0 minor:52 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/125be1ce61c5cbf87d6821fd3f6be8c59435587d70ab8313324e9f2d61ccffb1/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/56449eb052ecfe85a4cafce8de006036b05b1695841030b8ef177a5a4eb97b6a/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/007a56d57052ebfa79556119708d9b6cac4ca8a87935b1a1eda3093f03618d15/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/8569d0a4ca03468ec21153d8f3dea8f553487a9d894aa18ff114528f4e908534/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/113e186f876ee7ec71ccbc941515f329077f6cddc1791a0fc7e84a8e3451fc86/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/183b0e527b3e48498b22e89f005b5b1a69eeade031320aaf40b29ac609f4c305/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/038c286a82f7e581f50f5181f540fe7ececba7c7189c65527c7eb732955db22d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c2f3d4317350f7f1ee37c9a01bc4346835bb2283aa040b3ecfdfad365ac8d3e4/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/097e47a44cf68d4f07ed38143b980d457f83488a58b00a1615f85489f96c08be/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/d4ee730ba8959045fc2a8d4e4153b1559e05c1d2b5030b3900f5dd195498d05b/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/d2eaee7d08ee7aa133e84bd90d3e9e2f797b00a88e45725864de436c95e43fe9/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/67bbdf2e6ef97e340dc83cdf1c03cbfd6e082a2c770003b3973e06a42d874431/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/52a116da4d3e589f73b632dec1405f41368fd5945b8ea7e7b1647cf16c6775d1/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/40180a993e3d1e197b0f10b892a969d383c7fd52b8e7066a29b01e69d6397eb3/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/edd8d095e11c87ed6127bc4980d4748575f05d0b02d33c264df84e8679a298ed/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/1dab51cd628e6a94eb936c09530f0136252f228969e1ca69cd86e7d5bbe61936/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/a673aa4aca7e935480edada56a056f95ffdbba64f50befdc05a1d18de7afbfff/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e742df388662565dc552709556cc771e9168cc385bcbf6ed7e80f8742fe18191/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/3d70ede331539d6ca6a81e9fa2dde9fd83e8250cff108f6e835d406c02811309/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/06d8675c3b3588369d7b21eb25dee6af06d3ccd56713d98fe318f42d5082427a/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/2cc5b71fb118c455fd8ac6502df920bb46927f27b121ae4bf62311727b812585/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-202:{mountpoint:/var/lib/containers/storage/overlay/8b8a65d64190f827ac87c041f3124d2db2db9af8775c16f007f04cd785953276/merged major:0 minor:202 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/20b545469c28d9601b6613e56dde34191976160f981e57bf8c847e39deeb282a/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/faf6ab2dbe1deb917eb4cb29eaefd048867188417510ba0c7ce8f948745e2b1f/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/bd4751e6b6a436689b8fce23df69c9f9029635c22243451f9217abc77a2cc58e/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/28355646eaae17c881f1ddb70db6b884440e013fb34e97076159016e4991c89a/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/d9d1e895be67acdaa7e40ed86e05b94f73c056feff494fecddb6e023917dd08a/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/74fb774030688a3e43f40de9094c9a88868c1094fabd74f81631af2e5403fc52/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/87b03b3b274ce94543c2683dfc26114d2752836150d590130a2b2285a510048b/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/5c5aae0715bf469e6afdf4425858f3c7bbfbaf22ee0abcc79763ee6c0202e6fa/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/124749144ddd21523e4144cb786620da20fc0ebf524f9679d88115c5b916fc6e/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/9e13ef79f3e97c8108c1b42e58ec41f3fff0ee0b3ef2f5e0d80917cd78d6ffff/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/d4f60da019426f5e41de4fc4c7d8b2bfaa3d07dcb201dbcfa0ebdcb501e1d320/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/2502aaf2c50ebbd85b526e413aa7ec6973c0c6fa59d2275e2464246ff4d93d23/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/8a00cd25dcbf1b7e28db4723f45ae293354b25604c08fda63f562c0926ad3a1f/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/1c5f39fa271dd907f3f7afeba435b1e3a16e7bb264c759acc74d5bbdd00d0ee1/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/1863a6c33c035ce0b5a61dfbc9bb211f10a9b965338ea6134f51e2dab7556c0e/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/24e76b13ee518db385a715d810aecd221954e33be09640ad0244c22e5d271452/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/03f5c52277680edf2c450ad0264dbe94f3ca2138ac564a70543df88bba9c5526/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/d4f9d7ba823d96009712d978758343bc63a2d9fd9301b66d1060570a1ede0f38/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/125d809942eb4a1031e2c3f2665f55f3a739febfc20238bdd21a1f093a5df367/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/29ca8818eeddc270f8a13827941c24e0bb73d1579fc0e185467c72207b3732af/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/379a5398f0545cdbe6e8c6cefab5f2ce62f7274093eb4c2452e5c588e28ba031/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/e52aaafa0fdcf9dbdc79fcbe64a3c36243ab84adef1cfce19fc5cd9c8ee99997/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/528a195f758e0410caff83a460fc277d4a4b5ec8eb16db220abb47efe3b447e9/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/09dcdf63975c7dc0fbc51c83ad88572c2f4e43071f820fef52c85fcf0cb5b4ac/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/b1a8280d639f7677d3aae8747058b39c43dc5cac3446a6e52184574a71fc2e51/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-69:{mountpoint:/var/lib/containers/storage/overlay/79a78d46f470c68bfadcf68a831274a567f7ad46506be94675fa12b683fb2873/merged major:0 minor:69 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/fd83c38d52a83de33e4339f8ad4458b240aad839a967f957041f1a3d8afbd271/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-73:{mountpoint:/var/lib/containers/storage/overlay/e32cc2618e3813e8a8268071f430df380106ae8599aba64a6b899369d26dda77/merged major:0 minor:73 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/83744a76d37d4c0b7745d78088ccec0933639956ea36f9f5994f3c6a29fbca7b/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/6d9132e872e34a4879d21e8210831346275f88c19fa8af601c994615c2c9aff0/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/74f9d301d615e0ebdfb682efd33094285999dfe5693e25aa628fdff08cb331af/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/57f9af11dbf62921b7a41a894dcd3152942d761e1bf64396ea532a6b1f8c55fa/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/8f5b0aa51d518c069650a7213fdc5b7c6d614d04a622656a64526407e63aa104/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/a6b3fc8e9e7b31075f6ce3c0e3dd2226260c3280d4e628db87dcb631c6afe5d7/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/9ce91d26ba62e7143ae3952b1acac76cf1bfa2a3c5eb70e3eb0ca6ecdeae32b2/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 22:48:44.736010 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.730743 5114 manager.go:217] Machine: {Timestamp:2026-11-02 22:48:44.730232468 +0000 UTC m=+0.083265834 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3186440 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21efe55406fefe399efc598815a689 SystemUUID:ec21efe5-5406-fefe-399e-fc598815a689 BootID:2dab55aa-a2ba-4c2d-851e-5da4b2003631 Filesystems:[{Device:/var/lib/kubelet/pods/5ce08d97-9151-4a82-92b3-a01a4057f046/volumes/kubernetes.io~projected/kube-api-access-qpfqn DeviceMajor:0 DeviceMinor:54 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-73 DeviceMajor:0 DeviceMinor:73 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9887e0ad9b044eb9cca02058d29958f530800fbc5768f2445a8c7f229b674f76/userdata/shm DeviceMajor:0 DeviceMinor:197 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b77d42eb9ea696f91f01f98fa13c0571e8ecbc3d8d5e0039236a7fe32d46c502/userdata/shm DeviceMajor:0 DeviceMinor:277 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0c414822-2ac4-4cd7-844f-a1f334dba25a/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:332 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8fd672dc0cc08aad560279c4603970941fb1575285bcbea70018da387a77c42a/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90ca4505d28c37a4e751200f1e0076d676713d8b2963645e83c2e23b434563e6/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e608a3bc-a7f2-45a1-9603-d85bbbe20614/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:298 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/39117eeba9a2f4d86737c81b31806feef362b7e62df1d7fb6acf06b9904ba182/userdata/shm DeviceMajor:0 DeviceMinor:303 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/d0121629-a394-4f12-a90c-2f4246436bc5/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:217 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d/volumes/kubernetes.io~projected/kube-api-access-77psj DeviceMajor:0 DeviceMinor:336 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/94071069-300b-43de-87e3-58e31ac8ab9d/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:283 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee507e21-156a-48a5-9c73-5f9e98bdd44a/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/73682abc-fb01-4ac2-a073-fcffad51a512/volumes/kubernetes.io~projected/kube-api-access-nnvrv DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4468bd03a890b8fae2c6d143db0a84a5e10601fc853fd2ae6e9c4c61b5481300/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:334 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e95567ad1efeb49460dea7e0e951b3c0c82b7739e5d86b3220a03cb0f25a531/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b6048231-1f5f-482a-af32-228a7c75f910/volumes/kubernetes.io~projected/kube-api-access-m7zxb DeviceMajor:0 DeviceMinor:294 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca56f427-3734-48ec-b7ce-100daeb7b3bb/volumes/kubernetes.io~projected/kube-api-access-pcgff DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a6201c52a6605c5c878d4d4889bec6411be08be8372537ce6b6e5fc1c29f333/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db5f685d-3887-4e83-99f8-7538f184224f/volumes/kubernetes.io~projected/kube-api-access-tr6fc DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d05df5bf-8c7c-4c79-afa6-96fc7d3689e4/volumes/kubernetes.io~projected/kube-api-access-cm5cb DeviceMajor:0 DeviceMinor:53 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0e82120d-ed55-489b-a3d5-a553ac1b1678/volumes/kubernetes.io~projected/kube-api-access-sj75b DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc5798118300a384cf4aba07c07566f07b52867f51baaf40da43e1569a39264c/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ee507e21-156a-48a5-9c73-5f9e98bdd44a/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0a5e437f-4589-4f76-b777-026186f06c73/volumes/kubernetes.io~projected/kube-api-access-66b2k DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d00480a202d2e0f59c4e97f43655a9ebc9742638459c8d5eac352dc1c64aee5/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0121629-a394-4f12-a90c-2f4246436bc5/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad69acdc-e65a-4aad-ac33-354f6ea07401/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:273 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c8a3a269fc77b0500375f98669c75aa984112b8933bf283336a1eafd429f027/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db5f685d-3887-4e83-99f8-7538f184224f/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e82120d-ed55-489b-a3d5-a553ac1b1678/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/d0121629-a394-4f12-a90c-2f4246436bc5/volumes/kubernetes.io~projected/kube-api-access-5mh6p DeviceMajor:0 DeviceMinor:220 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af5ad537206c6aee77ca429dbccb84de80c1cd0c900448f27ec20ef813a5f69d/userdata/shm DeviceMajor:0 DeviceMinor:339 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1891a2fee619b7749bd605bce66982e33fdf968cd5798254a682a85b5afe31a2/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d0121629-a394-4f12-a90c-2f4246436bc5/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:215 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31f0641478a464484c9f4d8b85a2165de817c41bc124000b33749a28401bba05/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b113915b35fee03e088cb22493cd9621f8a66cce5eb9036eb4ccde329bbf16ff/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c414822-2ac4-4cd7-844f-a1f334dba25a/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:328 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00d4ccbee98a6deedb9d504bf767e5a7a9cf38c0c1f85f8ba00911ad35ff1ab6/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-202 DeviceMajor:0 DeviceMinor:202 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c414822-2ac4-4cd7-844f-a1f334dba25a/volumes/kubernetes.io~projected/kube-api-access-gw64f DeviceMajor:0 DeviceMinor:335 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1322aa24-24c5-4a2e-b767-9bda95b4daaa/volumes/kubernetes.io~projected/kube-api-access-rl98f DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/754fedf886b43368b928ff4a1d861296eb55a85d000114da6a782f4f41fbdf89/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-69 DeviceMajor:0 DeviceMinor:69 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4/volumes/kubernetes.io~projected/kube-api-access-kxxkn DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3eff20b4dd4839ad801ecc8d1fd85f714de2dc60127bd547b37c0912bfafb0ef/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e93ae72ab7af8d8f988c835df9607896d50473144bc74bfb5a4d8038e771102d/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/901a3af424db738e9f6f250dd427c12d5a46087d7b5623159f66f7c229fd230f/userdata/shm DeviceMajor:0 DeviceMinor:337 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/32856eec-17a2-4d3a-8ca1-53755398f81c/volumes/kubernetes.io~projected/kube-api-access-pg6gf DeviceMajor:0 DeviceMinor:51 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d746a2fbc6bc62606a9cdbe5304b36eed622f947615dd7572e7fd331e4ef8cb/userdata/shm DeviceMajor:0 DeviceMinor:299 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fabb0266-226c-4935-a9fc-5ed6199e2f9b/volumes/kubernetes.io~projected/kube-api-access-9lppg DeviceMajor:0 DeviceMinor:52 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1322aa24-24c5-4a2e-b767-9bda95b4daaa/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/50a8014b3378c3ec8a909389a05a913ec59d4a9b30c4a80f1efd93bdceceaf01/userdata/shm DeviceMajor:0 DeviceMinor:295 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/1062888a-e10c-4a34-b84a-f85ff9eb8063/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/38ed00fd-8810-4f2d-868e-6ab2edcb860e/volumes/kubernetes.io~projected/kube-api-access-t6655 DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:333 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0e82120d-ed55-489b-a3d5-a553ac1b1678/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/38ed00fd-8810-4f2d-868e-6ab2edcb860e/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:216 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ce08d97-9151-4a82-92b3-a01a4057f046/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:297 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee507e21-156a-48a5-9c73-5f9e98bdd44a/volumes/kubernetes.io~projected/kube-api-access-6j2w5 DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94071069-300b-43de-87e3-58e31ac8ab9d/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:288 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/94071069-300b-43de-87e3-58e31ac8ab9d/volumes/kubernetes.io~projected/kube-api-access-pw7mm DeviceMajor:0 DeviceMinor:289 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d0121629-a394-4f12-a90c-2f4246436bc5/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:214 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:00d4ccbee98a6de MacAddress:d2:0c:e9:66:9c:a7 Speed:10000 Mtu:8901} {Name:2c8a3a269fc77b0 MacAddress:f6:8d:24:3b:eb:9c Speed:10000 Mtu:8901} {Name:2d746a2fbc6bc62 MacAddress:0e:99:48:f7:7d:ba Speed:10000 Mtu:8901} {Name:31f0641478a4644 MacAddress:56:80:a6:21:84:9a Speed:10000 Mtu:8901} {Name:39117eeba9a2f4d MacAddress:32:c0:bb:06:bd:68 Speed:10000 Mtu:8901} {Name:50a8014b3378c3e MacAddress:ee:79:aa:22:89:a4 Speed:10000 Mtu:8901} {Name:901a3af424db738 MacAddress:ea:41:16:e6:28:ae Speed:10000 Mtu:8901} {Name:9887e0ad9b044eb MacAddress:7e:d6:44:27:47:ae Speed:10000 Mtu:8901} {Name:9e95567ad1efeb4 MacAddress:2e:23:cc:96:bf:5e Speed:10000 Mtu:8901} {Name:b113915b35fee03 MacAddress:1a:ff:8e:41:60:a6 Speed:10000 Mtu:8901} {Name:b77d42eb9ea696f MacAddress:0a:b2:3e:b0:67:ba Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:47:1b:b2:82:95 Speed:0 Mtu:9001} {Name:br-int MacAddress:be:84:a9:45:20:9a Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:47:1b:b2:82:95 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:3e:67:8b:40:95:5c Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:a2:d4:93:73:8d:b2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 22:48:44.736010 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.731112 5114 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 22:48:44.736010 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.731171 5114 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 22:48:44.736010 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.731462 5114 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 22:48:44.736010 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.731505 5114 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-250.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 22:48:44.736010 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.731661 5114 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 22:48:44.736010 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.731670 5114 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 22:48:44.736010 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.731683 5114 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:48:44.736010 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.731699 5114 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:48:44.736010 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.731817 5114 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 22:48:44.736010 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.731996 5114 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 22:48:44.736010 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.732118 5114 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 22:48:44.736010 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.732156 5114 kubelet.go:493] "Attempting to sync node with API server" Nov 02 22:48:44.736010 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.732166 5114 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 22:48:44.736010 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.732177 5114 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 22:48:44.736010 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.732187 5114 kubelet.go:405] "Adding apiserver pod source" Nov 02 22:48:44.736010 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.732196 5114 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 22:48:44.736010 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.732966 5114 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 22:48:44.736010 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.732977 5114 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:48:44.736010 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.733620 5114 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 22:48:44.742936 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.742916 5114 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 22:48:44.742936 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.742936 5114 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:48:44.743061 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.743046 5114 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 22:48:44.743263 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.743252 5114 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 22:48:44.743293 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.743282 5114 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 22:48:44.743392 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.743384 5114 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 22:48:44.743427 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.743399 5114 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 22:48:44.743427 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.743406 5114 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 22:48:44.743427 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.743411 5114 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 22:48:44.743427 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.743417 5114 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 22:48:44.743427 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.743422 5114 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 22:48:44.743427 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.743428 5114 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 22:48:44.743622 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.743433 5114 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 22:48:44.743622 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.743440 5114 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 22:48:44.743622 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.743446 5114 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 22:48:44.743622 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.743454 5114 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 22:48:44.743622 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.743463 5114 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 22:48:44.743622 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.743511 5114 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 22:48:44.743622 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.743518 5114 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 22:48:44.747108 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.747092 5114 server.go:1263] "Started kubelet" Nov 02 22:48:44.747261 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.747202 5114 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 22:48:44.748516 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.747208 5114 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 22:48:44.748516 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.747543 5114 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 22:48:44.748516 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.747870 5114 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 22:48:44.747873 ip-10-0-135-250 systemd[1]: Started Kubernetes Kubelet. Nov 02 22:48:44.751034 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.749673 5114 server.go:317] "Adding debug handlers to kubelet server" Nov 02 22:48:44.752851 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.752829 5114 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 22:48:44.753190 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.753125 5114 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 22:48:44.753190 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.753190 5114 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.754863 5114 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:48:44.755037 5114 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-250.us-west-1.compute.internal\" not found" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.755081 5114 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.755187 5114 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 22:42:34 +0000 UTC" deadline="2028-08-03 08:45:26.18027677 +0000 UTC" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.755212 5114 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15345h56m41.425067657s" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.756288 5114 factory.go:153] Registering CRI-O factory Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.756304 5114 factory.go:223] Registration of the crio container factory successfully Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.756402 5114 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.756415 5114 factory.go:55] Registering systemd factory Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.756431 5114 factory.go:223] Registration of the systemd container factory successfully Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.756454 5114 factory.go:103] Registering Raw factory Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.756471 5114 manager.go:1199] Started watching for new ooms in manager Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.756897 5114 manager.go:319] Starting recovery of all containers Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758053 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a5e437f-4589-4f76-b777-026186f06c73" volumeName="kubernetes.io/configmap/0a5e437f-4589-4f76-b777-026186f06c73-cni-binary-copy" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758094 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a5e437f-4589-4f76-b777-026186f06c73" volumeName="kubernetes.io/projected/0a5e437f-4589-4f76-b777-026186f06c73-kube-api-access-66b2k" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758110 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e82120d-ed55-489b-a3d5-a553ac1b1678" volumeName="kubernetes.io/configmap/0e82120d-ed55-489b-a3d5-a553ac1b1678-ovnkube-script-lib" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758123 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad69acdc-e65a-4aad-ac33-354f6ea07401" volumeName="kubernetes.io/secret/ad69acdc-e65a-4aad-ac33-354f6ea07401-tls-certificates" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758137 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d" volumeName="kubernetes.io/secret/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-tls" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758189 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca56f427-3734-48ec-b7ce-100daeb7b3bb" volumeName="kubernetes.io/projected/ca56f427-3734-48ec-b7ce-100daeb7b3bb-kube-api-access-pcgff" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758205 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0121629-a394-4f12-a90c-2f4246436bc5" volumeName="kubernetes.io/configmap/d0121629-a394-4f12-a90c-2f4246436bc5-trusted-ca" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758215 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5f685d-3887-4e83-99f8-7538f184224f" volumeName="kubernetes.io/secret/db5f685d-3887-4e83-99f8-7538f184224f-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758235 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ce08d97-9151-4a82-92b3-a01a4057f046" volumeName="kubernetes.io/projected/5ce08d97-9151-4a82-92b3-a01a4057f046-kube-api-access-qpfqn" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758248 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6048231-1f5f-482a-af32-228a7c75f910" volumeName="kubernetes.io/empty-dir/b6048231-1f5f-482a-af32-228a7c75f910-tmp" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758262 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d" volumeName="kubernetes.io/secret/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758295 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73682abc-fb01-4ac2-a073-fcffad51a512" volumeName="kubernetes.io/configmap/73682abc-fb01-4ac2-a073-fcffad51a512-serviceca" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758309 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94071069-300b-43de-87e3-58e31ac8ab9d" volumeName="kubernetes.io/secret/94071069-300b-43de-87e3-58e31ac8ab9d-prometheus-operator-tls" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758323 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee507e21-156a-48a5-9c73-5f9e98bdd44a" volumeName="kubernetes.io/empty-dir/ee507e21-156a-48a5-9c73-5f9e98bdd44a-tmp" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758342 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c414822-2ac4-4cd7-844f-a1f334dba25a" volumeName="kubernetes.io/projected/0c414822-2ac4-4cd7-844f-a1f334dba25a-kube-api-access-gw64f" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758357 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0121629-a394-4f12-a90c-2f4246436bc5" volumeName="kubernetes.io/projected/d0121629-a394-4f12-a90c-2f4246436bc5-bound-sa-token" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758373 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0121629-a394-4f12-a90c-2f4246436bc5" volumeName="kubernetes.io/secret/d0121629-a394-4f12-a90c-2f4246436bc5-image-registry-private-configuration" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758390 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c414822-2ac4-4cd7-844f-a1f334dba25a" volumeName="kubernetes.io/configmap/0c414822-2ac4-4cd7-844f-a1f334dba25a-metrics-client-ca" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758402 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94071069-300b-43de-87e3-58e31ac8ab9d" volumeName="kubernetes.io/secret/94071069-300b-43de-87e3-58e31ac8ab9d-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758416 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6048231-1f5f-482a-af32-228a7c75f910" volumeName="kubernetes.io/projected/b6048231-1f5f-482a-af32-228a7c75f910-kube-api-access-m7zxb" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758433 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d05df5bf-8c7c-4c79-afa6-96fc7d3689e4" volumeName="kubernetes.io/projected/d05df5bf-8c7c-4c79-afa6-96fc7d3689e4-kube-api-access-cm5cb" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758447 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32856eec-17a2-4d3a-8ca1-53755398f81c" volumeName="kubernetes.io/projected/32856eec-17a2-4d3a-8ca1-53755398f81c-kube-api-access-pg6gf" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758460 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d" volumeName="kubernetes.io/configmap/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-metrics-client-ca" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758472 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0121629-a394-4f12-a90c-2f4246436bc5" volumeName="kubernetes.io/projected/d0121629-a394-4f12-a90c-2f4246436bc5-registry-tls" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758514 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5f685d-3887-4e83-99f8-7538f184224f" volumeName="kubernetes.io/empty-dir/db5f685d-3887-4e83-99f8-7538f184224f-data-volume" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758526 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c414822-2ac4-4cd7-844f-a1f334dba25a" volumeName="kubernetes.io/empty-dir/0c414822-2ac4-4cd7-844f-a1f334dba25a-volume-directive-shadow" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758543 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0121629-a394-4f12-a90c-2f4246436bc5" volumeName="kubernetes.io/projected/d0121629-a394-4f12-a90c-2f4246436bc5-kube-api-access-5mh6p" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758580 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0121629-a394-4f12-a90c-2f4246436bc5" volumeName="kubernetes.io/secret/d0121629-a394-4f12-a90c-2f4246436bc5-installation-pull-secrets" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758599 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5f685d-3887-4e83-99f8-7538f184224f" volumeName="kubernetes.io/configmap/db5f685d-3887-4e83-99f8-7538f184224f-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758614 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e608a3bc-a7f2-45a1-9603-d85bbbe20614" volumeName="kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758631 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e82120d-ed55-489b-a3d5-a553ac1b1678" volumeName="kubernetes.io/secret/0e82120d-ed55-489b-a3d5-a553ac1b1678-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758644 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1062888a-e10c-4a34-b84a-f85ff9eb8063" volumeName="kubernetes.io/configmap/1062888a-e10c-4a34-b84a-f85ff9eb8063-konnectivity-ca" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758658 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ce08d97-9151-4a82-92b3-a01a4057f046" volumeName="kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758674 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94071069-300b-43de-87e3-58e31ac8ab9d" volumeName="kubernetes.io/projected/94071069-300b-43de-87e3-58e31ac8ab9d-kube-api-access-pw7mm" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758688 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5f685d-3887-4e83-99f8-7538f184224f" volumeName="kubernetes.io/projected/db5f685d-3887-4e83-99f8-7538f184224f-kube-api-access-tr6fc" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758702 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8acfea0-a4d3-4673-9fc6-e48a94dfbba4" volumeName="kubernetes.io/projected/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4-kube-api-access-kxxkn" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758716 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1062888a-e10c-4a34-b84a-f85ff9eb8063" volumeName="kubernetes.io/secret/1062888a-e10c-4a34-b84a-f85ff9eb8063-agent-certs" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758731 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38ed00fd-8810-4f2d-868e-6ab2edcb860e" volumeName="kubernetes.io/empty-dir/38ed00fd-8810-4f2d-868e-6ab2edcb860e-tmp-dir" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758744 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0121629-a394-4f12-a90c-2f4246436bc5" volumeName="kubernetes.io/empty-dir/d0121629-a394-4f12-a90c-2f4246436bc5-ca-trust-extracted" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758756 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e82120d-ed55-489b-a3d5-a553ac1b1678" volumeName="kubernetes.io/configmap/0e82120d-ed55-489b-a3d5-a553ac1b1678-ovnkube-config" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758767 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e82120d-ed55-489b-a3d5-a553ac1b1678" volumeName="kubernetes.io/projected/0e82120d-ed55-489b-a3d5-a553ac1b1678-kube-api-access-sj75b" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758782 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1322aa24-24c5-4a2e-b767-9bda95b4daaa" volumeName="kubernetes.io/projected/1322aa24-24c5-4a2e-b767-9bda95b4daaa-kube-api-access-rl98f" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758797 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32856eec-17a2-4d3a-8ca1-53755398f81c" volumeName="kubernetes.io/configmap/32856eec-17a2-4d3a-8ca1-53755398f81c-iptables-alerter-script" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758808 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38ed00fd-8810-4f2d-868e-6ab2edcb860e" volumeName="kubernetes.io/configmap/38ed00fd-8810-4f2d-868e-6ab2edcb860e-config-volume" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758822 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38ed00fd-8810-4f2d-868e-6ab2edcb860e" volumeName="kubernetes.io/projected/38ed00fd-8810-4f2d-868e-6ab2edcb860e-kube-api-access-t6655" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758833 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73682abc-fb01-4ac2-a073-fcffad51a512" volumeName="kubernetes.io/projected/73682abc-fb01-4ac2-a073-fcffad51a512-kube-api-access-nnvrv" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758845 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94071069-300b-43de-87e3-58e31ac8ab9d" volumeName="kubernetes.io/configmap/94071069-300b-43de-87e3-58e31ac8ab9d-metrics-client-ca" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758856 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a5e437f-4589-4f76-b777-026186f06c73" volumeName="kubernetes.io/configmap/0a5e437f-4589-4f76-b777-026186f06c73-multus-daemon-config" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758869 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1322aa24-24c5-4a2e-b767-9bda95b4daaa" volumeName="kubernetes.io/secret/1322aa24-24c5-4a2e-b767-9bda95b4daaa-cert" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758886 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d" volumeName="kubernetes.io/empty-dir/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-textfile" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758903 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d" volumeName="kubernetes.io/projected/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-kube-api-access-77psj" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758917 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c414822-2ac4-4cd7-844f-a1f334dba25a" volumeName="kubernetes.io/secret/0c414822-2ac4-4cd7-844f-a1f334dba25a-kube-state-metrics-tls" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758932 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca56f427-3734-48ec-b7ce-100daeb7b3bb" volumeName="kubernetes.io/configmap/ca56f427-3734-48ec-b7ce-100daeb7b3bb-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758947 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca56f427-3734-48ec-b7ce-100daeb7b3bb" volumeName="kubernetes.io/configmap/ca56f427-3734-48ec-b7ce-100daeb7b3bb-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758960 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0121629-a394-4f12-a90c-2f4246436bc5" volumeName="kubernetes.io/configmap/d0121629-a394-4f12-a90c-2f4246436bc5-registry-certificates" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758973 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee507e21-156a-48a5-9c73-5f9e98bdd44a" volumeName="kubernetes.io/empty-dir/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-tuned" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.758994 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e82120d-ed55-489b-a3d5-a553ac1b1678" volumeName="kubernetes.io/configmap/0e82120d-ed55-489b-a3d5-a553ac1b1678-env-overrides" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.759007 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee507e21-156a-48a5-9c73-5f9e98bdd44a" volumeName="kubernetes.io/projected/ee507e21-156a-48a5-9c73-5f9e98bdd44a-kube-api-access-6j2w5" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.759018 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c414822-2ac4-4cd7-844f-a1f334dba25a" volumeName="kubernetes.io/configmap/0c414822-2ac4-4cd7-844f-a1f334dba25a-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.759029 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d05df5bf-8c7c-4c79-afa6-96fc7d3689e4" volumeName="kubernetes.io/empty-dir/d05df5bf-8c7c-4c79-afa6-96fc7d3689e4-tmp-dir" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.759043 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c414822-2ac4-4cd7-844f-a1f334dba25a" volumeName="kubernetes.io/secret/0c414822-2ac4-4cd7-844f-a1f334dba25a-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.759054 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38ed00fd-8810-4f2d-868e-6ab2edcb860e" volumeName="kubernetes.io/secret/38ed00fd-8810-4f2d-868e-6ab2edcb860e-metrics-tls" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.759066 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca56f427-3734-48ec-b7ce-100daeb7b3bb" volumeName="kubernetes.io/configmap/ca56f427-3734-48ec-b7ce-100daeb7b3bb-cni-binary-copy" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.759076 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d" volumeName="kubernetes.io/configmap/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.759088 5114 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fabb0266-226c-4935-a9fc-5ed6199e2f9b" volumeName="kubernetes.io/projected/fabb0266-226c-4935-a9fc-5ed6199e2f9b-kube-api-access-9lppg" seLinuxMountContext="" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.759100 5114 reconstruct.go:97] "Volume reconstruction finished" Nov 02 22:48:44.759510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.759108 5114 reconciler.go:29] "Reconciler: start to sync state" Nov 02 22:48:44.784416 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.784372 5114 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 22:48:44.785877 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.785850 5114 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 22:48:44.785877 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.785875 5114 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 22:48:44.786116 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.785901 5114 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 22:48:44.786116 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:48:44.785938 5114 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 22:48:44.801401 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.801353 5114 generic.go:356] "Generic (PLEG): container finished" podID="ca56f427-3734-48ec-b7ce-100daeb7b3bb" containerID="4635195381cdb42ab8c319701accac85d0deeb3bbc6a71f0eb2e8f161ec03dc9" exitCode=0 Nov 02 22:48:44.801401 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.801392 5114 generic.go:356] "Generic (PLEG): container finished" podID="ca56f427-3734-48ec-b7ce-100daeb7b3bb" containerID="1eea70647448f7afcd915c40434937e611db4aa6cd456e23330f2e2f4476035a" exitCode=0 Nov 02 22:48:44.801401 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.801399 5114 generic.go:356] "Generic (PLEG): container finished" podID="ca56f427-3734-48ec-b7ce-100daeb7b3bb" containerID="b2d264c1057858855976df1e2b843c740e925d38f9a8de51af78738734e48086" exitCode=0 Nov 02 22:48:44.801401 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.801405 5114 generic.go:356] "Generic (PLEG): container finished" podID="ca56f427-3734-48ec-b7ce-100daeb7b3bb" containerID="253e35fdd9b1b91d69fffb9897a1e25a724d5695b87e492f4d6ff3af8a79939c" exitCode=0 Nov 02 22:48:44.801401 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.801410 5114 generic.go:356] "Generic (PLEG): container finished" podID="ca56f427-3734-48ec-b7ce-100daeb7b3bb" containerID="426023fab7f19abe44365501205abdbc26ebc64395d885c789ea721f45725f74" exitCode=0 Nov 02 22:48:44.801401 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.801415 5114 generic.go:356] "Generic (PLEG): container finished" podID="ca56f427-3734-48ec-b7ce-100daeb7b3bb" containerID="fcb10083c0bba9d367fdb7df95b93b7af2b34efb9fa61d2ecad9527722b49c2a" exitCode=0 Nov 02 22:48:44.811374 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.811326 5114 generic.go:356] "Generic (PLEG): container finished" podID="7c6e8e95c46679d654a23b98c776960b" containerID="6f0c8168b3e8c62fa7327bf821e4d7c471d15249703638dd6f8b60a082211254" exitCode=0 Nov 02 22:48:44.846656 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.846590 5114 manager.go:324] Recovery completed Nov 02 22:48:44.857569 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:48:44.855513 5114 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-250.us-west-1.compute.internal\" not found" Nov 02 22:48:44.865674 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.865651 5114 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:48:44.867868 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.867841 5114 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:48:44.867975 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.867884 5114 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:48:44.867975 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.867898 5114 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:48:44.870577 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.870561 5114 cpu_manager.go:226] "Starting" policy="none" Nov 02 22:48:44.870577 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.870572 5114 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:48:44.870662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.870576 5114 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 22:48:44.870662 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.870648 5114 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 22:48:44.870854 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.870803 5114 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 22:48:44.870901 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.870856 5114 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 22:48:44.870931 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.870903 5114 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 22:48:44.870931 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.870910 5114 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 22:48:44.870931 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.870915 5114 policy_none.go:50] "Start" Nov 02 22:48:44.870931 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.870927 5114 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 22:48:44.871035 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.870938 5114 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 22:48:44.871085 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.871075 5114 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 22:48:44.871114 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.871087 5114 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 22:48:44.871114 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.871093 5114 policy_none.go:44] "Start" Nov 02 22:48:44.871529 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.871511 5114 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:48:44.871626 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.871547 5114 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:48:44.871626 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.871562 5114 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:48:44.875540 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.875511 5114 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 22:48:44.878291 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.878277 5114 manager.go:341] "Starting Device Plugin manager" Nov 02 22:48:44.878374 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:48:44.878306 5114 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 22:48:44.878374 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.878316 5114 server.go:87] "Starting device plugin registration server" Nov 02 22:48:44.878694 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.878680 5114 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 22:48:44.878749 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.878697 5114 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 22:48:44.878815 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.878766 5114 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 22:48:44.878914 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.878873 5114 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 22:48:44.879033 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.878921 5114 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 22:48:44.879033 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.878928 5114 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 22:48:44.879137 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.878987 5114 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T22:48:44.878909425Z","UUID":"5392ce29-a08f-4499-9bb5-338f98eb266b","Handler":null,"Name":"","Endpoint":""} Nov 02 22:48:44.880454 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.880434 5114 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 22:48:44.880652 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.880465 5114 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 22:48:44.880652 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:48:44.880547 5114 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 22:48:44.880652 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:48:44.880589 5114 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-250.us-west-1.compute.internal\" not found" Nov 02 22:48:44.886103 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.886055 5114 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal"] Nov 02 22:48:44.886248 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.886219 5114 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:48:44.887114 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.887095 5114 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:48:44.887208 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.887129 5114 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:48:44.887208 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.887146 5114 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:48:44.887549 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.887533 5114 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:48:44.887652 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.887636 5114 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:48:44.888349 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.888333 5114 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:48:44.888442 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.888349 5114 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:48:44.888442 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.888362 5114 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:48:44.888442 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.888370 5114 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:48:44.888442 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.888379 5114 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:48:44.888442 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.888386 5114 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:48:44.888710 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:48:44.888634 5114 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-250.us-west-1.compute.internal\" not found" node="ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:48:44.888809 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.888772 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal" event={"ID":"09bea7df1640e9a4e82191575067a028","Type":"ContainerStarted","Data":"8c9e0bb9a88617f90d531a1ff935b24f82d3062f8a2dfe088706577583f6eb0b"} Nov 02 22:48:44.888863 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.888824 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal" event={"ID":"09bea7df1640e9a4e82191575067a028","Type":"ContainerStarted","Data":"8fd672dc0cc08aad560279c4603970941fb1575285bcbea70018da387a77c42a"} Nov 02 22:48:44.888917 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.888882 5114 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:48:44.888917 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.888887 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" event={"ID":"7c6e8e95c46679d654a23b98c776960b","Type":"ContainerStarted","Data":"5d43e1f41298e458edf771adba427272cbf4d9b461d3e732975317b3f66e656c"} Nov 02 22:48:44.888917 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.888910 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" event={"ID":"7c6e8e95c46679d654a23b98c776960b","Type":"ContainerDied","Data":"6f0c8168b3e8c62fa7327bf821e4d7c471d15249703638dd6f8b60a082211254"} Nov 02 22:48:44.889059 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.888930 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" event={"ID":"7c6e8e95c46679d654a23b98c776960b","Type":"ContainerStarted","Data":"fc5798118300a384cf4aba07c07566f07b52867f51baaf40da43e1569a39264c"} Nov 02 22:48:44.889669 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.889651 5114 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:48:44.889777 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.889686 5114 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:48:44.889777 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.889702 5114 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:48:44.889888 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:48:44.889873 5114 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-250.us-west-1.compute.internal\" not found" node="ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:48:44.942651 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.942611 5114 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:48:44.942651 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.942622 5114 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:48:44.943045 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.943030 5114 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 22:48:44.943437 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.943422 5114 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:48:44.955717 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.955682 5114 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:48:44.978836 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.978796 5114 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:48:44.980004 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.979980 5114 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:48:44.980107 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.980017 5114 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:48:44.980107 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.980027 5114 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:48:44.980107 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:44.980069 5114 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:48:45.040978 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.040941 5114 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:48:45.041697 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.041668 5114 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:48:45.041831 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:48:45.041710 5114 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:48:45.041831 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.041723 5114 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:48:45.043899 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.043875 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09bea7df1640e9a4e82191575067a028-config\") pod \"kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal\" (UID: \"09bea7df1640e9a4e82191575067a028\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:48:45.043990 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.043915 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c6e8e95c46679d654a23b98c776960b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal\" (UID: \"7c6e8e95c46679d654a23b98c776960b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:48:45.043990 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.043940 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c6e8e95c46679d654a23b98c776960b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal\" (UID: \"7c6e8e95c46679d654a23b98c776960b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:48:45.094726 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.094692 5114 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:48:45.094873 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.094782 5114 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:48:45.095411 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.095394 5114 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeNotReady" Nov 02 22:48:45.095475 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.095429 5114 setters.go:546] "Node became not ready" node="ip-10-0-135-250.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-02T22:48:45Z","lastTransitionTime":"2026-11-02T22:48:45Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 02 22:48:45.115593 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.115505 5114 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:48:45.115593 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:48:45.115562 5114 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:48:45.144658 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.144628 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c6e8e95c46679d654a23b98c776960b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal\" (UID: \"7c6e8e95c46679d654a23b98c776960b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:48:45.144751 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.144662 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c6e8e95c46679d654a23b98c776960b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal\" (UID: \"7c6e8e95c46679d654a23b98c776960b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:48:45.144751 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.144681 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09bea7df1640e9a4e82191575067a028-config\") pod \"kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal\" (UID: \"09bea7df1640e9a4e82191575067a028\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:48:45.144751 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.144731 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09bea7df1640e9a4e82191575067a028-config\") pod \"kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal\" (UID: \"09bea7df1640e9a4e82191575067a028\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:48:45.144919 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.144754 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c6e8e95c46679d654a23b98c776960b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal\" (UID: \"7c6e8e95c46679d654a23b98c776960b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:48:45.144919 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.144755 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c6e8e95c46679d654a23b98c776960b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal\" (UID: \"7c6e8e95c46679d654a23b98c776960b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:48:45.160659 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.160617 5114 nodeinfomanager.go:471] existing CSINode "ip-10-0-135-250.us-west-1.compute.internal" is owned by different node (oldNodeID="ca08649d-db38-4ccf-8054-fb799c46d546", newNodeID=""), cleaning up... Nov 02 22:48:45.732638 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.732592 5114 apiserver.go:52] "Watching apiserver" Nov 02 22:48:45.835698 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.835666 5114 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 22:48:45.836294 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.836269 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-hg78p","openshift-network-operator/iptables-alerter-jvpc5","openshift-multus/network-metrics-daemon-fqwl4","kube-system/konnectivity-agent-ftp5v","openshift-ingress-canary/ingress-canary-vtndb","openshift-monitoring/node-exporter-pw6nq","openshift-dns/node-resolver-l5tth","openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2","openshift-monitoring/prometheus-operator-9cd89b74c-p266m","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8lzdt","kube-system/global-pull-secret-syncer-9drjv","openshift-dns/dns-default-mzgfc","openshift-network-diagnostics/network-check-target-569mb","kube-system/kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal","openshift-console/downloads-6dd789c8b7-vtccr","openshift-image-registry/image-registry-8bdc9bc8d-tcg5t","openshift-image-registry/node-ca-dsp96","openshift-insights/insights-runtime-extractor-8qzjx","openshift-ovn-kubernetes/ovnkube-node-d25g6","openshift-cluster-node-tuning-operator/tuned-9pf69","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-lpqcq"] Nov 02 22:48:45.844920 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.844891 5114 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:48:45.845158 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.845140 5114 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:48:45.856580 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.856547 5114 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 22:48:45.890566 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.890526 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 22:48:45.890566 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.890522 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 22:48:45.890566 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.890540 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 22:48:45.890944 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.890540 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 22:48:45.890944 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.890556 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 22:48:45.890944 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.890603 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 22:48:45.890944 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.890611 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mb4zk\"" Nov 02 22:48:45.890944 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.890625 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 22:48:45.890944 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.890656 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 22:48:45.890944 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.890675 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 22:48:45.890944 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.890710 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 22:48:45.890944 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.890734 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 22:48:45.890944 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.890764 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:48:45.890944 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.890751 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pl59x\"" Nov 02 22:48:45.890944 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.890798 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 22:48:45.890944 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.890902 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-77tzt\"" Nov 02 22:48:45.890944 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.890905 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 22:48:45.890944 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.890947 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 22:48:45.891406 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.890977 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 22:48:45.891406 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.890987 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9jqgm\"" Nov 02 22:48:45.891406 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.891029 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hnmtx\"" Nov 02 22:48:45.891406 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.891063 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 22:48:45.891406 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.891124 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 22:48:45.891406 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.891183 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r8wwr\"" Nov 02 22:48:45.891406 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.891371 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 22:48:45.891631 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.891599 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f4f84\"" Nov 02 22:48:45.891903 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.891891 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 22:48:45.892683 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.892669 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 22:48:45.892849 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.892836 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 22:48:45.893121 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.893109 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 22:48:45.893190 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.893180 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bxrdd\"" Nov 02 22:48:45.893246 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.893234 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 22:48:45.899127 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.899105 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 22:48:45.899179 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.899108 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:48:45.899256 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.899242 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zfnv8\"" Nov 02 22:48:45.899531 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.899501 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 22:48:45.899531 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.899519 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wk6k8\"" Nov 02 22:48:45.899628 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.899579 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9g8qt\"" Nov 02 22:48:45.899908 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.899890 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 22:48:45.900071 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.900052 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 22:48:45.900151 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.900059 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 22:48:45.900151 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.900104 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 22:48:45.900151 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.900149 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d58gp\"" Nov 02 22:48:45.900300 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.900148 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 22:48:45.900300 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.900193 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 22:48:45.900300 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.900236 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qf97t\"" Nov 02 22:48:45.900424 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.900371 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 22:48:45.900424 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.900415 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p2dkz\"" Nov 02 22:48:45.900553 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.900465 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 22:48:45.900553 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.900513 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 22:48:45.940993 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.940945 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 22:48:45.944902 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.944875 5114 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:48:45.945018 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:48:45.944939 5114 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:48:45.947911 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.947893 5114 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:48:45.947996 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:48:45.947980 5114 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-250.us-west-1.compute.internal" Nov 02 22:48:45.950231 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.950215 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ed00fd-8810-4f2d-868e-6ab2edcb860e-tmp-dir\") pod \"dns-default-mzgfc\" (UID: \"38ed00fd-8810-4f2d-868e-6ab2edcb860e\") " pod="openshift-dns/dns-default-mzgfc" Nov 02 22:48:45.950288 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.950243 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-sys-fs\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:48:45.950288 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.950266 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:45.950288 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.950283 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66b2k\" (UniqueName: \"kubernetes.io/projected/0a5e437f-4589-4f76-b777-026186f06c73-kube-api-access-66b2k\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:45.950380 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.950299 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77psj\" (UniqueName: \"kubernetes.io/projected/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-kube-api-access-77psj\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:45.950380 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.950335 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-etc-openvswitch\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:45.950380 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.950357 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-log-socket\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:45.950380 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.950378 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-run-k8s-cni-cncf-io\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:45.950539 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.950404 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ed00fd-8810-4f2d-868e-6ab2edcb860e-tmp-dir\") pod \"dns-default-mzgfc\" (UID: \"38ed00fd-8810-4f2d-868e-6ab2edcb860e\") " pod="openshift-dns/dns-default-mzgfc" Nov 02 22:48:45.950539 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.950460 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1062888a-e10c-4a34-b84a-f85ff9eb8063-konnectivity-ca\") pod \"konnectivity-agent-ftp5v\" (UID: \"1062888a-e10c-4a34-b84a-f85ff9eb8063\") " pod="kube-system/konnectivity-agent-ftp5v" Nov 02 22:48:45.950539 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.950513 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/73682abc-fb01-4ac2-a073-fcffad51a512-serviceca\") pod \"node-ca-dsp96\" (UID: \"73682abc-fb01-4ac2-a073-fcffad51a512\") " pod="openshift-image-registry/node-ca-dsp96" Nov 02 22:48:45.950661 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.950620 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj75b\" (UniqueName: \"kubernetes.io/projected/0e82120d-ed55-489b-a3d5-a553ac1b1678-kube-api-access-sj75b\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:45.950693 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.950657 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1062888a-e10c-4a34-b84a-f85ff9eb8063-konnectivity-ca\") pod \"konnectivity-agent-ftp5v\" (UID: \"1062888a-e10c-4a34-b84a-f85ff9eb8063\") " pod="kube-system/konnectivity-agent-ftp5v" Nov 02 22:48:45.950721 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.950702 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0121629-a394-4f12-a90c-2f4246436bc5-trusted-ca\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:45.950749 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.950727 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a5e437f-4589-4f76-b777-026186f06c73-cni-binary-copy\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:45.950781 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.950750 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-lib-modules\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:45.950871 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.950854 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-tuned\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:45.950924 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.950887 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db5f685d-3887-4e83-99f8-7538f184224f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:45.950996 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.950960 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-wtmp\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:45.950996 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.950981 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-tuned\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:45.951075 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951041 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0121629-a394-4f12-a90c-2f4246436bc5-trusted-ca\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:45.951075 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951053 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a5e437f-4589-4f76-b777-026186f06c73-cni-binary-copy\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:45.951075 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951062 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-kubelet\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:45.951169 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951099 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-hostroot\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:45.951169 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951126 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e82120d-ed55-489b-a3d5-a553ac1b1678-ovn-node-metrics-cert\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:45.951169 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951165 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73682abc-fb01-4ac2-a073-fcffad51a512-host\") pod \"node-ca-dsp96\" (UID: \"73682abc-fb01-4ac2-a073-fcffad51a512\") " pod="openshift-image-registry/node-ca-dsp96" Nov 02 22:48:45.951273 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951183 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-sysctl-d\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:45.951273 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951205 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7zxb\" (UniqueName: \"kubernetes.io/projected/b6048231-1f5f-482a-af32-228a7c75f910-kube-api-access-m7zxb\") pod \"downloads-6dd789c8b7-vtccr\" (UID: \"b6048231-1f5f-482a-af32-228a7c75f910\") " pod="openshift-console/downloads-6dd789c8b7-vtccr" Nov 02 22:48:45.951273 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951211 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/73682abc-fb01-4ac2-a073-fcffad51a512-serviceca\") pod \"node-ca-dsp96\" (UID: \"73682abc-fb01-4ac2-a073-fcffad51a512\") " pod="openshift-image-registry/node-ca-dsp96" Nov 02 22:48:45.951273 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951249 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-run-multus-certs\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:45.951468 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951276 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-socket-dir\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:48:45.951468 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951302 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-device-dir\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:48:45.951468 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951349 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c414822-2ac4-4cd7-844f-a1f334dba25a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:45.951468 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951363 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e82120d-ed55-489b-a3d5-a553ac1b1678-ovn-node-metrics-cert\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:45.951468 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951396 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-tls\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:45.951468 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951451 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs\") pod \"network-metrics-daemon-fqwl4\" (UID: \"5ce08d97-9151-4a82-92b3-a01a4057f046\") " pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:48:45.951829 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951536 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca56f427-3734-48ec-b7ce-100daeb7b3bb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:48:45.951829 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951570 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-multus-socket-dir-parent\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:45.951829 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951597 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-multus-conf-dir\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:45.951829 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951624 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr6fc\" (UniqueName: \"kubernetes.io/projected/db5f685d-3887-4e83-99f8-7538f184224f-kube-api-access-tr6fc\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:45.951829 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951644 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-textfile\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:45.951829 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951680 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-host\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:45.951829 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951710 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-textfile\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:45.951829 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951716 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ce08d97-9151-4a82-92b3-a01a4057f046-metrics-certs\") pod \"network-metrics-daemon-fqwl4\" (UID: \"5ce08d97-9151-4a82-92b3-a01a4057f046\") " pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:48:45.951829 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951745 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret\") pod \"global-pull-secret-syncer-9drjv\" (UID: \"e608a3bc-a7f2-45a1-9603-d85bbbe20614\") " pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:48:45.951829 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951771 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-var-lib-cni-bin\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:45.951829 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951786 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-var-lib-cni-multus\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:45.951829 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951812 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6655\" (UniqueName: \"kubernetes.io/projected/38ed00fd-8810-4f2d-868e-6ab2edcb860e-kube-api-access-t6655\") pod \"dns-default-mzgfc\" (UID: \"38ed00fd-8810-4f2d-868e-6ab2edcb860e\") " pod="openshift-dns/dns-default-mzgfc" Nov 02 22:48:45.952320 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951845 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca56f427-3734-48ec-b7ce-100daeb7b3bb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:48:45.952320 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951879 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/db5f685d-3887-4e83-99f8-7538f184224f-data-volume\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:45.952320 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951912 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpfqn\" (UniqueName: \"kubernetes.io/projected/5ce08d97-9151-4a82-92b3-a01a4057f046-kube-api-access-qpfqn\") pod \"network-metrics-daemon-fqwl4\" (UID: \"5ce08d97-9151-4a82-92b3-a01a4057f046\") " pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:48:45.952320 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951929 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-modprobe-d\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:45.952320 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951944 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0121629-a394-4f12-a90c-2f4246436bc5-registry-certificates\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:45.952320 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951961 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0121629-a394-4f12-a90c-2f4246436bc5-bound-sa-token\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:45.952320 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.951990 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca56f427-3734-48ec-b7ce-100daeb7b3bb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:48:45.952320 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952051 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/db5f685d-3887-4e83-99f8-7538f184224f-data-volume\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:45.952320 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952066 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94071069-300b-43de-87e3-58e31ac8ab9d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-p266m\" (UID: \"94071069-300b-43de-87e3-58e31ac8ab9d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" Nov 02 22:48:45.952320 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952074 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e608a3bc-a7f2-45a1-9603-d85bbbe20614-original-pull-secret\") pod \"global-pull-secret-syncer-9drjv\" (UID: \"e608a3bc-a7f2-45a1-9603-d85bbbe20614\") " pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:48:45.952320 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952108 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-var-lib-openvswitch\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:45.952320 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952139 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32856eec-17a2-4d3a-8ca1-53755398f81c-host-slash\") pod \"iptables-alerter-jvpc5\" (UID: \"32856eec-17a2-4d3a-8ca1-53755398f81c\") " pod="openshift-network-operator/iptables-alerter-jvpc5" Nov 02 22:48:45.952320 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952171 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca56f427-3734-48ec-b7ce-100daeb7b3bb-system-cni-dir\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:48:45.952320 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952182 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0121629-a394-4f12-a90c-2f4246436bc5-registry-certificates\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:45.952320 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952200 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1322aa24-24c5-4a2e-b767-9bda95b4daaa-cert\") pod \"ingress-canary-vtndb\" (UID: \"1322aa24-24c5-4a2e-b767-9bda95b4daaa\") " pod="openshift-ingress-canary/ingress-canary-vtndb" Nov 02 22:48:45.952320 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952218 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-sysconfig\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:45.952320 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952232 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-sysctl-conf\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:45.952320 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952281 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-etc-selinux\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:48:45.952320 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952313 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lppg\" (UniqueName: \"kubernetes.io/projected/fabb0266-226c-4935-a9fc-5ed6199e2f9b-kube-api-access-9lppg\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:48:45.952916 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952361 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0c414822-2ac4-4cd7-844f-a1f334dba25a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:45.952916 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952469 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-run-netns\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:45.952916 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952540 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0c414822-2ac4-4cd7-844f-a1f334dba25a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:45.952916 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952579 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0121629-a394-4f12-a90c-2f4246436bc5-installation-pull-secrets\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:45.952916 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952603 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm5cb\" (UniqueName: \"kubernetes.io/projected/d05df5bf-8c7c-4c79-afa6-96fc7d3689e4-kube-api-access-cm5cb\") pod \"node-resolver-l5tth\" (UID: \"d05df5bf-8c7c-4c79-afa6-96fc7d3689e4\") " pod="openshift-dns/node-resolver-l5tth" Nov 02 22:48:45.952916 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952667 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnvrv\" (UniqueName: \"kubernetes.io/projected/73682abc-fb01-4ac2-a073-fcffad51a512-kube-api-access-nnvrv\") pod \"node-ca-dsp96\" (UID: \"73682abc-fb01-4ac2-a073-fcffad51a512\") " pod="openshift-image-registry/node-ca-dsp96" Nov 02 22:48:45.952916 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952685 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-kubernetes\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:45.952916 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952700 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c414822-2ac4-4cd7-844f-a1f334dba25a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:45.952916 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952715 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-run-openvswitch\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:45.952916 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952732 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-cni-bin\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:45.952916 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952757 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0121629-a394-4f12-a90c-2f4246436bc5-installation-pull-secrets\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:45.952916 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952775 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-system-cni-dir\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:45.952916 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952875 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-etc-kubernetes\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:45.952916 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952901 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw64f\" (UniqueName: \"kubernetes.io/projected/0c414822-2ac4-4cd7-844f-a1f334dba25a-kube-api-access-gw64f\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:45.952916 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952920 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-var-lib-kubelet\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:45.953565 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.952974 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca56f427-3734-48ec-b7ce-100daeb7b3bb-cnibin\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:48:45.953565 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953074 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:48:45.953565 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953107 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c414822-2ac4-4cd7-844f-a1f334dba25a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:45.953565 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953128 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0c414822-2ac4-4cd7-844f-a1f334dba25a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:45.953565 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953172 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0e82120d-ed55-489b-a3d5-a553ac1b1678-ovnkube-script-lib\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:45.953565 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953208 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mh6p\" (UniqueName: \"kubernetes.io/projected/d0121629-a394-4f12-a90c-2f4246436bc5-kube-api-access-5mh6p\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:45.953565 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953315 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl98f\" (UniqueName: \"kubernetes.io/projected/1322aa24-24c5-4a2e-b767-9bda95b4daaa-kube-api-access-rl98f\") pod \"ingress-canary-vtndb\" (UID: \"1322aa24-24c5-4a2e-b767-9bda95b4daaa\") " pod="openshift-ingress-canary/ingress-canary-vtndb" Nov 02 22:48:45.953565 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953332 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee507e21-156a-48a5-9c73-5f9e98bdd44a-tmp\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:45.953565 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953349 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38ed00fd-8810-4f2d-868e-6ab2edcb860e-metrics-tls\") pod \"dns-default-mzgfc\" (UID: \"38ed00fd-8810-4f2d-868e-6ab2edcb860e\") " pod="openshift-dns/dns-default-mzgfc" Nov 02 22:48:45.953565 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953365 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxxkn\" (UniqueName: \"kubernetes.io/projected/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4-kube-api-access-kxxkn\") pod \"network-check-target-569mb\" (UID: \"f8acfea0-a4d3-4673-9fc6-e48a94dfbba4\") " pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:48:45.953565 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953365 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0e82120d-ed55-489b-a3d5-a553ac1b1678-ovnkube-script-lib\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:45.953565 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953384 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-run-systemd\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:45.953565 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953415 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-node-log\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:45.953565 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953456 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee507e21-156a-48a5-9c73-5f9e98bdd44a-tmp\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:45.953565 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953465 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-multus-cni-dir\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953582 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38ed00fd-8810-4f2d-868e-6ab2edcb860e-metrics-tls\") pod \"dns-default-mzgfc\" (UID: \"38ed00fd-8810-4f2d-868e-6ab2edcb860e\") " pod="openshift-dns/dns-default-mzgfc" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953594 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca56f427-3734-48ec-b7ce-100daeb7b3bb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953630 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e608a3bc-a7f2-45a1-9603-d85bbbe20614-kubelet-config\") pod \"global-pull-secret-syncer-9drjv\" (UID: \"e608a3bc-a7f2-45a1-9603-d85bbbe20614\") " pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953658 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953685 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0e82120d-ed55-489b-a3d5-a553ac1b1678-ovnkube-config\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953738 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-run-netns\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953762 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-run\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953764 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca56f427-3734-48ec-b7ce-100daeb7b3bb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953811 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0121629-a394-4f12-a90c-2f4246436bc5-ca-trust-extracted\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953834 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw7mm\" (UniqueName: \"kubernetes.io/projected/94071069-300b-43de-87e3-58e31ac8ab9d-kube-api-access-pw7mm\") pod \"prometheus-operator-9cd89b74c-p266m\" (UID: \"94071069-300b-43de-87e3-58e31ac8ab9d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953857 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32856eec-17a2-4d3a-8ca1-53755398f81c-iptables-alerter-script\") pod \"iptables-alerter-jvpc5\" (UID: \"32856eec-17a2-4d3a-8ca1-53755398f81c\") " pod="openshift-network-operator/iptables-alerter-jvpc5" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953872 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0e82120d-ed55-489b-a3d5-a553ac1b1678-ovnkube-config\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953874 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg6gf\" (UniqueName: \"kubernetes.io/projected/32856eec-17a2-4d3a-8ca1-53755398f81c-kube-api-access-pg6gf\") pod \"iptables-alerter-jvpc5\" (UID: \"32856eec-17a2-4d3a-8ca1-53755398f81c\") " pod="openshift-network-operator/iptables-alerter-jvpc5" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953909 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d05df5bf-8c7c-4c79-afa6-96fc7d3689e4-hosts-file\") pod \"node-resolver-l5tth\" (UID: \"d05df5bf-8c7c-4c79-afa6-96fc7d3689e4\") " pod="openshift-dns/node-resolver-l5tth" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953936 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38ed00fd-8810-4f2d-868e-6ab2edcb860e-config-volume\") pod \"dns-default-mzgfc\" (UID: \"38ed00fd-8810-4f2d-868e-6ab2edcb860e\") " pod="openshift-dns/dns-default-mzgfc" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953946 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0121629-a394-4f12-a90c-2f4246436bc5-ca-trust-extracted\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.953960 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-run-ovn\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954004 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0e82120d-ed55-489b-a3d5-a553ac1b1678-env-overrides\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954025 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32856eec-17a2-4d3a-8ca1-53755398f81c-iptables-alerter-script\") pod \"iptables-alerter-jvpc5\" (UID: \"32856eec-17a2-4d3a-8ca1-53755398f81c\") " pod="openshift-network-operator/iptables-alerter-jvpc5" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954061 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d05df5bf-8c7c-4c79-afa6-96fc7d3689e4-tmp-dir\") pod \"node-resolver-l5tth\" (UID: \"d05df5bf-8c7c-4c79-afa6-96fc7d3689e4\") " pod="openshift-dns/node-resolver-l5tth" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954088 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e608a3bc-a7f2-45a1-9603-d85bbbe20614-dbus\") pod \"global-pull-secret-syncer-9drjv\" (UID: \"e608a3bc-a7f2-45a1-9603-d85bbbe20614\") " pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954134 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0e82120d-ed55-489b-a3d5-a553ac1b1678-env-overrides\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954152 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d05df5bf-8c7c-4c79-afa6-96fc7d3689e4-tmp-dir\") pod \"node-resolver-l5tth\" (UID: \"d05df5bf-8c7c-4c79-afa6-96fc7d3689e4\") " pod="openshift-dns/node-resolver-l5tth" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954174 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38ed00fd-8810-4f2d-868e-6ab2edcb860e-config-volume\") pod \"dns-default-mzgfc\" (UID: \"38ed00fd-8810-4f2d-868e-6ab2edcb860e\") " pod="openshift-dns/dns-default-mzgfc" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954186 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/94071069-300b-43de-87e3-58e31ac8ab9d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-p266m\" (UID: \"94071069-300b-43de-87e3-58e31ac8ab9d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" Nov 02 22:48:45.954214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954224 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca56f427-3734-48ec-b7ce-100daeb7b3bb-cni-binary-copy\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:48:45.954968 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954243 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ad69acdc-e65a-4aad-ac33-354f6ea07401-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-8lzdt\" (UID: \"ad69acdc-e65a-4aad-ac33-354f6ea07401\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8lzdt" Nov 02 22:48:45.954968 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954262 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-systemd-units\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:45.954968 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954278 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6048231-1f5f-482a-af32-228a7c75f910-tmp\") pod \"downloads-6dd789c8b7-vtccr\" (UID: \"b6048231-1f5f-482a-af32-228a7c75f910\") " pod="openshift-console/downloads-6dd789c8b7-vtccr" Nov 02 22:48:45.954968 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954331 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-systemd\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:45.954968 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954346 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6048231-1f5f-482a-af32-228a7c75f910-tmp\") pod \"downloads-6dd789c8b7-vtccr\" (UID: \"b6048231-1f5f-482a-af32-228a7c75f910\") " pod="openshift-console/downloads-6dd789c8b7-vtccr" Nov 02 22:48:45.954968 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954393 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j2w5\" (UniqueName: \"kubernetes.io/projected/ee507e21-156a-48a5-9c73-5f9e98bdd44a-kube-api-access-6j2w5\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:45.954968 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954420 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca56f427-3734-48ec-b7ce-100daeb7b3bb-cni-binary-copy\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:48:45.954968 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954422 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-cni-netd\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:45.954968 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954459 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94071069-300b-43de-87e3-58e31ac8ab9d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-p266m\" (UID: \"94071069-300b-43de-87e3-58e31ac8ab9d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" Nov 02 22:48:45.954968 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954525 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-var-lib-kubelet\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:45.954968 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954560 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0121629-a394-4f12-a90c-2f4246436bc5-image-registry-private-configuration\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:45.954968 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954601 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db5f685d-3887-4e83-99f8-7538f184224f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:45.954968 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954669 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-sys\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:45.954968 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954700 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-accelerators-collector-config\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:45.954968 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954725 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-cnibin\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:45.954968 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954755 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1062888a-e10c-4a34-b84a-f85ff9eb8063-agent-certs\") pod \"konnectivity-agent-ftp5v\" (UID: \"1062888a-e10c-4a34-b84a-f85ff9eb8063\") " pod="kube-system/konnectivity-agent-ftp5v" Nov 02 22:48:45.954968 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954765 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0121629-a394-4f12-a90c-2f4246436bc5-image-registry-private-configuration\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:45.954968 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954784 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-registration-dir\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:48:45.954968 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954882 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-metrics-client-ca\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:45.954968 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954940 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-slash\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:45.954968 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954961 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1062888a-e10c-4a34-b84a-f85ff9eb8063-agent-certs\") pod \"konnectivity-agent-ftp5v\" (UID: \"1062888a-e10c-4a34-b84a-f85ff9eb8063\") " pod="kube-system/konnectivity-agent-ftp5v" Nov 02 22:48:45.954968 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.954973 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-sys\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:45.955579 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.955011 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/db5f685d-3887-4e83-99f8-7538f184224f-crio-socket\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:45.955579 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.955031 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-root\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:45.955579 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.955052 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-run-ovn-kubernetes\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:45.955579 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.955069 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0121629-a394-4f12-a90c-2f4246436bc5-registry-tls\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:45.955579 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.955096 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a5e437f-4589-4f76-b777-026186f06c73-multus-daemon-config\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:45.955579 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.955112 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcgff\" (UniqueName: \"kubernetes.io/projected/ca56f427-3734-48ec-b7ce-100daeb7b3bb-kube-api-access-pcgff\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:48:45.955579 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.955225 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0121629-a394-4f12-a90c-2f4246436bc5-registry-tls\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:45.955579 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.955295 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a5e437f-4589-4f76-b777-026186f06c73-multus-daemon-config\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:45.990637 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.990550 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 22:48:45.990637 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.990565 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 22:48:45.990637 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.990582 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 22:48:45.990815 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.990657 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 22:48:45.990815 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.990790 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 22:48:45.990815 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.990795 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 22:48:45.990971 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.990841 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 22:48:45.990971 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.990872 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 22:48:45.990971 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.990908 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 22:48:45.990971 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.990913 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 22:48:45.990971 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.990935 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 22:48:45.991239 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.990985 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 22:48:45.991239 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.990985 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 22:48:45.991239 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.991007 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 22:48:45.991239 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.991013 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 22:48:45.991239 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.991050 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bd5jt\"" Nov 02 22:48:45.991239 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.991080 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 02 22:48:45.991239 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.991107 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 22:48:45.991239 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.991111 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5z48j\"" Nov 02 22:48:45.991239 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.991113 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 22:48:45.991239 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.991126 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-cqvbp\"" Nov 02 22:48:45.991239 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.991134 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-scv9s\"" Nov 02 22:48:45.991239 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.991172 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-v2s7h\"" Nov 02 22:48:45.991710 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.991289 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 22:48:45.991710 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.991319 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db5f685d-3887-4e83-99f8-7538f184224f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:45.991710 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.991349 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 22:48:45.991710 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.991323 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 22:48:45.991710 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.991403 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-cz5vj\"" Nov 02 22:48:45.991710 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.991438 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-467lc\"" Nov 02 22:48:45.991710 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.991584 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c414822-2ac4-4cd7-844f-a1f334dba25a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:45.991893 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.991746 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-tls\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:45.992425 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.992407 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94071069-300b-43de-87e3-58e31ac8ab9d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-p266m\" (UID: \"94071069-300b-43de-87e3-58e31ac8ab9d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" Nov 02 22:48:45.992504 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.992474 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1322aa24-24c5-4a2e-b767-9bda95b4daaa-cert\") pod \"ingress-canary-vtndb\" (UID: \"1322aa24-24c5-4a2e-b767-9bda95b4daaa\") " pod="openshift-ingress-canary/ingress-canary-vtndb" Nov 02 22:48:45.993653 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.993637 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c414822-2ac4-4cd7-844f-a1f334dba25a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:45.993728 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.993662 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0c414822-2ac4-4cd7-844f-a1f334dba25a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:45.993728 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.993667 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c414822-2ac4-4cd7-844f-a1f334dba25a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:45.994804 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.994787 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db5f685d-3887-4e83-99f8-7538f184224f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:45.994859 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.994799 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ad69acdc-e65a-4aad-ac33-354f6ea07401-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-8lzdt\" (UID: \"ad69acdc-e65a-4aad-ac33-354f6ea07401\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8lzdt" Nov 02 22:48:45.994859 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.994808 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94071069-300b-43de-87e3-58e31ac8ab9d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-p266m\" (UID: \"94071069-300b-43de-87e3-58e31ac8ab9d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" Nov 02 22:48:45.994859 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.994828 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/94071069-300b-43de-87e3-58e31ac8ab9d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-p266m\" (UID: \"94071069-300b-43de-87e3-58e31ac8ab9d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" Nov 02 22:48:45.994940 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.994903 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:45.994940 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.994929 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-accelerators-collector-config\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:45.995073 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:45.995060 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-metrics-client-ca\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:46.010949 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.010923 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr6fc\" (UniqueName: \"kubernetes.io/projected/db5f685d-3887-4e83-99f8-7538f184224f-kube-api-access-tr6fc\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:46.011024 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.010928 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66b2k\" (UniqueName: \"kubernetes.io/projected/0a5e437f-4589-4f76-b777-026186f06c73-kube-api-access-66b2k\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.011126 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.011110 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj75b\" (UniqueName: \"kubernetes.io/projected/0e82120d-ed55-489b-a3d5-a553ac1b1678-kube-api-access-sj75b\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.011747 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.011732 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0121629-a394-4f12-a90c-2f4246436bc5-bound-sa-token\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:46.012224 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.012208 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg6gf\" (UniqueName: \"kubernetes.io/projected/32856eec-17a2-4d3a-8ca1-53755398f81c-kube-api-access-pg6gf\") pod \"iptables-alerter-jvpc5\" (UID: \"32856eec-17a2-4d3a-8ca1-53755398f81c\") " pod="openshift-network-operator/iptables-alerter-jvpc5" Nov 02 22:48:46.012461 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.012439 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lppg\" (UniqueName: \"kubernetes.io/projected/fabb0266-226c-4935-a9fc-5ed6199e2f9b-kube-api-access-9lppg\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:48:46.012511 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.012445 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7zxb\" (UniqueName: \"kubernetes.io/projected/b6048231-1f5f-482a-af32-228a7c75f910-kube-api-access-m7zxb\") pod \"downloads-6dd789c8b7-vtccr\" (UID: \"b6048231-1f5f-482a-af32-228a7c75f910\") " pod="openshift-console/downloads-6dd789c8b7-vtccr" Nov 02 22:48:46.012638 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.012616 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mh6p\" (UniqueName: \"kubernetes.io/projected/d0121629-a394-4f12-a90c-2f4246436bc5-kube-api-access-5mh6p\") pod \"image-registry-8bdc9bc8d-tcg5t\" (UID: \"d0121629-a394-4f12-a90c-2f4246436bc5\") " pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:46.013366 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.013342 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnvrv\" (UniqueName: \"kubernetes.io/projected/73682abc-fb01-4ac2-a073-fcffad51a512-kube-api-access-nnvrv\") pod \"node-ca-dsp96\" (UID: \"73682abc-fb01-4ac2-a073-fcffad51a512\") " pod="openshift-image-registry/node-ca-dsp96" Nov 02 22:48:46.014332 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.014312 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm5cb\" (UniqueName: \"kubernetes.io/projected/d05df5bf-8c7c-4c79-afa6-96fc7d3689e4-kube-api-access-cm5cb\") pod \"node-resolver-l5tth\" (UID: \"d05df5bf-8c7c-4c79-afa6-96fc7d3689e4\") " pod="openshift-dns/node-resolver-l5tth" Nov 02 22:48:46.014426 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.014405 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77psj\" (UniqueName: \"kubernetes.io/projected/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-kube-api-access-77psj\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:46.014923 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.014908 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl98f\" (UniqueName: \"kubernetes.io/projected/1322aa24-24c5-4a2e-b767-9bda95b4daaa-kube-api-access-rl98f\") pod \"ingress-canary-vtndb\" (UID: \"1322aa24-24c5-4a2e-b767-9bda95b4daaa\") " pod="openshift-ingress-canary/ingress-canary-vtndb" Nov 02 22:48:46.015941 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.015925 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw64f\" (UniqueName: \"kubernetes.io/projected/0c414822-2ac4-4cd7-844f-a1f334dba25a-kube-api-access-gw64f\") pod \"kube-state-metrics-5dc8d7d89b-sr5f2\" (UID: \"0c414822-2ac4-4cd7-844f-a1f334dba25a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" Nov 02 22:48:46.016017 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.016000 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcgff\" (UniqueName: \"kubernetes.io/projected/ca56f427-3734-48ec-b7ce-100daeb7b3bb-kube-api-access-pcgff\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:48:46.016147 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.016131 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j2w5\" (UniqueName: \"kubernetes.io/projected/ee507e21-156a-48a5-9c73-5f9e98bdd44a-kube-api-access-6j2w5\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:46.016835 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.016817 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpfqn\" (UniqueName: \"kubernetes.io/projected/5ce08d97-9151-4a82-92b3-a01a4057f046-kube-api-access-qpfqn\") pod \"network-metrics-daemon-fqwl4\" (UID: \"5ce08d97-9151-4a82-92b3-a01a4057f046\") " pod="openshift-multus/network-metrics-daemon-fqwl4" Nov 02 22:48:46.017310 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.017292 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6655\" (UniqueName: \"kubernetes.io/projected/38ed00fd-8810-4f2d-868e-6ab2edcb860e-kube-api-access-t6655\") pod \"dns-default-mzgfc\" (UID: \"38ed00fd-8810-4f2d-868e-6ab2edcb860e\") " pod="openshift-dns/dns-default-mzgfc" Nov 02 22:48:46.017354 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.017339 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw7mm\" (UniqueName: \"kubernetes.io/projected/94071069-300b-43de-87e3-58e31ac8ab9d-kube-api-access-pw7mm\") pod \"prometheus-operator-9cd89b74c-p266m\" (UID: \"94071069-300b-43de-87e3-58e31ac8ab9d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-p266m" Nov 02 22:48:46.019043 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.019027 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxxkn\" (UniqueName: \"kubernetes.io/projected/f8acfea0-a4d3-4673-9fc6-e48a94dfbba4-kube-api-access-kxxkn\") pod \"network-check-target-569mb\" (UID: \"f8acfea0-a4d3-4673-9fc6-e48a94dfbba4\") " pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:48:46.055866 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.055829 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-var-lib-cni-multus\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.055866 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.055869 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-modprobe-d\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:46.056067 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.055887 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-var-lib-openvswitch\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.056067 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.055940 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-var-lib-openvswitch\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.056067 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.055948 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-var-lib-cni-multus\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.056067 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.055969 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32856eec-17a2-4d3a-8ca1-53755398f81c-host-slash\") pod \"iptables-alerter-jvpc5\" (UID: \"32856eec-17a2-4d3a-8ca1-53755398f81c\") " pod="openshift-network-operator/iptables-alerter-jvpc5" Nov 02 22:48:46.056067 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.055994 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca56f427-3734-48ec-b7ce-100daeb7b3bb-system-cni-dir\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:48:46.056067 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.055999 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-modprobe-d\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:46.056067 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056059 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-sysconfig\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:46.056353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056075 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32856eec-17a2-4d3a-8ca1-53755398f81c-host-slash\") pod \"iptables-alerter-jvpc5\" (UID: \"32856eec-17a2-4d3a-8ca1-53755398f81c\") " pod="openshift-network-operator/iptables-alerter-jvpc5" Nov 02 22:48:46.056353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056084 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca56f427-3734-48ec-b7ce-100daeb7b3bb-system-cni-dir\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:48:46.056353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056097 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-sysctl-conf\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:46.056353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056123 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-etc-selinux\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:48:46.056353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056135 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-sysconfig\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:46.056353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056148 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-run-netns\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.056353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056176 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-kubernetes\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:46.056353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056182 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-etc-selinux\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:48:46.056353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056196 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-sysctl-conf\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:46.056353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056201 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-run-openvswitch\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.056353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056213 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-run-netns\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.056353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056225 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-cni-bin\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.056353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056226 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-kubernetes\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:46.056353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056254 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-run-openvswitch\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.056353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056260 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-cni-bin\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.056353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056251 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-system-cni-dir\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.056353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056294 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-etc-kubernetes\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.056353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056299 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-system-cni-dir\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.056353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056318 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-var-lib-kubelet\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:46.056353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056337 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-etc-kubernetes\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.056353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056341 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca56f427-3734-48ec-b7ce-100daeb7b3bb-cnibin\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056366 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056378 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca56f427-3734-48ec-b7ce-100daeb7b3bb-cnibin\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056387 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-run-systemd\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056384 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-var-lib-kubelet\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056418 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-run-systemd\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056425 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056452 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-node-log\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056470 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-multus-cni-dir\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056504 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e608a3bc-a7f2-45a1-9603-d85bbbe20614-kubelet-config\") pod \"global-pull-secret-syncer-9drjv\" (UID: \"e608a3bc-a7f2-45a1-9603-d85bbbe20614\") " pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056528 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-run-netns\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056545 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-run\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056545 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e608a3bc-a7f2-45a1-9603-d85bbbe20614-kubelet-config\") pod \"global-pull-secret-syncer-9drjv\" (UID: \"e608a3bc-a7f2-45a1-9603-d85bbbe20614\") " pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056552 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-multus-cni-dir\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056564 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d05df5bf-8c7c-4c79-afa6-96fc7d3689e4-hosts-file\") pod \"node-resolver-l5tth\" (UID: \"d05df5bf-8c7c-4c79-afa6-96fc7d3689e4\") " pod="openshift-dns/node-resolver-l5tth" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056588 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-run-ovn\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056591 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-run-netns\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056506 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-node-log\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056599 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-run\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056604 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e608a3bc-a7f2-45a1-9603-d85bbbe20614-dbus\") pod \"global-pull-secret-syncer-9drjv\" (UID: \"e608a3bc-a7f2-45a1-9603-d85bbbe20614\") " pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056638 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-run-ovn\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056646 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d05df5bf-8c7c-4c79-afa6-96fc7d3689e4-hosts-file\") pod \"node-resolver-l5tth\" (UID: \"d05df5bf-8c7c-4c79-afa6-96fc7d3689e4\") " pod="openshift-dns/node-resolver-l5tth" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056653 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e608a3bc-a7f2-45a1-9603-d85bbbe20614-dbus\") pod \"global-pull-secret-syncer-9drjv\" (UID: \"e608a3bc-a7f2-45a1-9603-d85bbbe20614\") " pod="kube-system/global-pull-secret-syncer-9drjv" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056690 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-systemd-units\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056706 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-systemd\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056724 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-systemd-units\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056746 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-cni-netd\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056754 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-systemd\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056764 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-var-lib-kubelet\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056788 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-cni-netd\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056797 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-var-lib-kubelet\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056816 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-sys\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056837 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-cnibin\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056853 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-registration-dir\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056878 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-slash\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056901 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-sys\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056920 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-registration-dir\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056924 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/db5f685d-3887-4e83-99f8-7538f184224f-crio-socket\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056920 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-cnibin\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056946 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-sys\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056949 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-slash\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056962 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-sys\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056956 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-root\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056989 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/db5f685d-3887-4e83-99f8-7538f184224f-crio-socket\") pod \"insights-runtime-extractor-8qzjx\" (UID: \"db5f685d-3887-4e83-99f8-7538f184224f\") " pod="openshift-insights/insights-runtime-extractor-8qzjx" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.056992 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-root\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057005 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-run-ovn-kubernetes\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057030 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-sys-fs\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057046 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057070 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-etc-openvswitch\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.057048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057084 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-log-socket\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057102 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-run-k8s-cni-cncf-io\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057100 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-sys-fs\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057107 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-run-ovn-kubernetes\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057128 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-etc-openvswitch\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057132 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-lib-modules\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057105 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057147 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-run-k8s-cni-cncf-io\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057158 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-wtmp\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057130 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-log-socket\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057179 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-kubelet\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057207 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e82120d-ed55-489b-a3d5-a553ac1b1678-host-kubelet\") pod \"ovnkube-node-d25g6\" (UID: \"0e82120d-ed55-489b-a3d5-a553ac1b1678\") " pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057232 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d-node-exporter-wtmp\") pod \"node-exporter-pw6nq\" (UID: \"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d\") " pod="openshift-monitoring/node-exporter-pw6nq" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057230 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-hostroot\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057267 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-hostroot\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057271 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73682abc-fb01-4ac2-a073-fcffad51a512-host\") pod \"node-ca-dsp96\" (UID: \"73682abc-fb01-4ac2-a073-fcffad51a512\") " pod="openshift-image-registry/node-ca-dsp96" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057279 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-lib-modules\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057295 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73682abc-fb01-4ac2-a073-fcffad51a512-host\") pod \"node-ca-dsp96\" (UID: \"73682abc-fb01-4ac2-a073-fcffad51a512\") " pod="openshift-image-registry/node-ca-dsp96" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057300 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-sysctl-d\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057328 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-run-multus-certs\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057353 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-socket-dir\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057379 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-device-dir\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057383 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-run-multus-certs\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057406 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca56f427-3734-48ec-b7ce-100daeb7b3bb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057431 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-multus-socket-dir-parent\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057410 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-etc-sysctl-d\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057450 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-socket-dir\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057458 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-multus-conf-dir\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057501 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fabb0266-226c-4935-a9fc-5ed6199e2f9b-device-dir\") pod \"aws-ebs-csi-driver-node-9fsp5\" (UID: \"fabb0266-226c-4935-a9fc-5ed6199e2f9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9fsp5" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057507 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-host\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057521 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-multus-socket-dir-parent\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057529 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca56f427-3734-48ec-b7ce-100daeb7b3bb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lpqcq\" (UID: \"ca56f427-3734-48ec-b7ce-100daeb7b3bb\") " pod="openshift-multus/multus-additional-cni-plugins-lpqcq" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057533 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-var-lib-cni-bin\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057545 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee507e21-156a-48a5-9c73-5f9e98bdd44a-host\") pod \"tuned-9pf69\" (UID: \"ee507e21-156a-48a5-9c73-5f9e98bdd44a\") " pod="openshift-cluster-node-tuning-operator/tuned-9pf69" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057563 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-host-var-lib-cni-bin\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.058552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.057527 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a5e437f-4589-4f76-b777-026186f06c73-multus-conf-dir\") pod \"multus-hg78p\" (UID: \"0a5e437f-4589-4f76-b777-026186f06c73\") " pod="openshift-multus/multus-hg78p" Nov 02 22:48:46.148953 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.148931 5114 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:48:46.505731 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.505694 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8lzdt" Nov 02 22:48:46.509308 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:46.509280 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8lzdt" Nov 02 22:48:47.797141 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:47.797102 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mzgfc" Nov 02 22:48:48.860323 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:48.860278 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" event={"ID":"0c414822-2ac4-4cd7-844f-a1f334dba25a","Type":"ContainerStarted","Data":"d721e87d3a14babae37a618317319fc04d1d30eced068e7d505f3a8b4d4a148c"} Nov 02 22:48:48.860938 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:48.860359 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" event={"ID":"0c414822-2ac4-4cd7-844f-a1f334dba25a","Type":"ContainerStarted","Data":"44ac09f9ec2573666476af7043159c3f7f4446531693542b3b0cb18b8e674f18"} Nov 02 22:48:48.860938 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:48.860386 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sr5f2" event={"ID":"0c414822-2ac4-4cd7-844f-a1f334dba25a","Type":"ContainerStarted","Data":"96290e8d2466a898861b082e8c254c149ea3f0aacadc5ef4144798bd841e3920"} Nov 02 22:48:48.863147 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:48.863114 5114 generic.go:356] "Generic (PLEG): container finished" podID="b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d" containerID="f5e59c6acb84bb45e0903b8f8298517e12b6955979f31fac483496c111e5d349" exitCode=0 Nov 02 22:48:48.863320 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:48.863189 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pw6nq" event={"ID":"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d","Type":"ContainerDied","Data":"f5e59c6acb84bb45e0903b8f8298517e12b6955979f31fac483496c111e5d349"} Nov 02 22:48:49.872006 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:49.871960 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pw6nq" event={"ID":"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d","Type":"ContainerStarted","Data":"aecc05aefcd373d7b173f5d2e2e85325f2f465734102c118f5cdd505ee10b683"} Nov 02 22:48:49.872444 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:49.872033 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pw6nq" event={"ID":"b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d","Type":"ContainerStarted","Data":"280ad024e0768e67f140af108bdcee258f449e08e965cbcf0a1d0b9c05ea296e"} Nov 02 22:48:50.414353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:50.414323 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-569mb" Nov 02 22:48:51.547225 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:51.547194 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:54.191689 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:54.191654 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8bdc9bc8d-tcg5t" Nov 02 22:48:54.642085 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:54.642048 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:48:54.642297 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:54.642278 5114 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 22:48:55.300759 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:48:55.300728 5114 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-250.us-west-1.compute.internal" event="NodeReady" Nov 02 22:49:00.554780 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.554740 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-759b69bf76-6phhk"] Nov 02 22:49:00.561167 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.560679 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:00.612829 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.612702 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 22:49:00.612829 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.612789 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 22:49:00.662423 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.662392 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 22:49:00.685520 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.685142 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-service-ca\") pod \"console-759b69bf76-6phhk\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:00.685520 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.685201 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b58306c-c5d7-4a3d-a65b-391c79964d11-tmp\") pod \"console-759b69bf76-6phhk\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:00.685520 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.685221 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxvkk\" (UniqueName: \"kubernetes.io/projected/0b58306c-c5d7-4a3d-a65b-391c79964d11-kube-api-access-dxvkk\") pod \"console-759b69bf76-6phhk\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:00.685520 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.685242 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-oauth-serving-cert\") pod \"console-759b69bf76-6phhk\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:00.685520 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.685282 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b58306c-c5d7-4a3d-a65b-391c79964d11-console-serving-cert\") pod \"console-759b69bf76-6phhk\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:00.685520 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.685297 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b58306c-c5d7-4a3d-a65b-391c79964d11-console-oauth-config\") pod \"console-759b69bf76-6phhk\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:00.685520 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.685324 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-console-config\") pod \"console-759b69bf76-6phhk\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:00.685520 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.685360 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-trusted-ca-bundle\") pod \"console-759b69bf76-6phhk\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:00.715305 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.715266 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 22:49:00.765270 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.765231 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 22:49:00.765561 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.765527 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kplp2\"" Nov 02 22:49:00.765869 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.765839 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 22:49:00.766380 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.766359 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-759b69bf76-6phhk"] Nov 02 22:49:00.786142 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.786104 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-trusted-ca-bundle\") pod \"console-759b69bf76-6phhk\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:00.786379 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.786160 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-service-ca\") pod \"console-759b69bf76-6phhk\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:00.786379 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.786205 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b58306c-c5d7-4a3d-a65b-391c79964d11-tmp\") pod \"console-759b69bf76-6phhk\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:00.786379 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.786231 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxvkk\" (UniqueName: \"kubernetes.io/projected/0b58306c-c5d7-4a3d-a65b-391c79964d11-kube-api-access-dxvkk\") pod \"console-759b69bf76-6phhk\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:00.786379 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.786258 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-oauth-serving-cert\") pod \"console-759b69bf76-6phhk\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:00.786379 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.786297 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b58306c-c5d7-4a3d-a65b-391c79964d11-console-serving-cert\") pod \"console-759b69bf76-6phhk\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:00.786379 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.786319 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b58306c-c5d7-4a3d-a65b-391c79964d11-console-oauth-config\") pod \"console-759b69bf76-6phhk\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:00.786379 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.786362 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-console-config\") pod \"console-759b69bf76-6phhk\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:00.787057 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.787033 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-service-ca\") pod \"console-759b69bf76-6phhk\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:00.787225 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.787190 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-trusted-ca-bundle\") pod \"console-759b69bf76-6phhk\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:00.787643 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.787619 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-console-config\") pod \"console-759b69bf76-6phhk\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:00.787970 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.787950 5114 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 22:49:00.788083 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.788064 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-oauth-serving-cert\") pod \"console-759b69bf76-6phhk\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:00.788346 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.788310 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b58306c-c5d7-4a3d-a65b-391c79964d11-tmp\") pod \"console-759b69bf76-6phhk\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:00.791163 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.791140 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b58306c-c5d7-4a3d-a65b-391c79964d11-console-serving-cert\") pod \"console-759b69bf76-6phhk\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:00.792244 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.792219 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b58306c-c5d7-4a3d-a65b-391c79964d11-console-oauth-config\") pod \"console-759b69bf76-6phhk\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:00.848422 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.848344 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxvkk\" (UniqueName: \"kubernetes.io/projected/0b58306c-c5d7-4a3d-a65b-391c79964d11-kube-api-access-dxvkk\") pod \"console-759b69bf76-6phhk\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:00.879278 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:00.879230 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:01.279315 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:01.279243 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-759b69bf76-6phhk"] Nov 02 22:49:01.282613 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:49:01.282581 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b58306c_c5d7_4a3d_a65b_391c79964d11.slice/crio-ebf8cb549b99f5f51955320a3c8bfcfd7843bf472c2783d271f2c61c569392e0 WatchSource:0}: Error finding container ebf8cb549b99f5f51955320a3c8bfcfd7843bf472c2783d271f2c61c569392e0: Status 404 returned error can't find the container with id ebf8cb549b99f5f51955320a3c8bfcfd7843bf472c2783d271f2c61c569392e0 Nov 02 22:49:01.922933 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:01.922893 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-759b69bf76-6phhk" event={"ID":"0b58306c-c5d7-4a3d-a65b-391c79964d11","Type":"ContainerStarted","Data":"ebf8cb549b99f5f51955320a3c8bfcfd7843bf472c2783d271f2c61c569392e0"} Nov 02 22:49:02.563984 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:02.563954 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ftp5v" Nov 02 22:49:05.942041 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:05.942002 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-759b69bf76-6phhk" event={"ID":"0b58306c-c5d7-4a3d-a65b-391c79964d11","Type":"ContainerStarted","Data":"08f4fcff5b59fcdb09385e292ad93f569fccd4c05a47485958f10e9848938a9e"} Nov 02 22:49:06.061170 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:06.060802 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-759b69bf76-6phhk" podStartSLOduration=2.469172172 podStartE2EDuration="6.060788435s" totalImagesPullingTime="3.591616263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:49:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:49:06.060428208 +0000 UTC m=+21.413461583" watchObservedRunningTime="2026-11-02 22:49:06.060788435 +0000 UTC m=+21.413821854" Nov 02 22:49:07.590138 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:49:07.590088 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 22:49:07.880899 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:07.880826 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d25g6" Nov 02 22:49:07.951948 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:07.951909 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-vtccr" event={"ID":"b6048231-1f5f-482a-af32-228a7c75f910","Type":"ContainerStarted","Data":"ec0154ca5cca55ae4c60b704ad8c04a0c54e6c9b879832b896f71f1f2df31f75"} Nov 02 22:49:07.952232 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:07.952208 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-vtccr" Nov 02 22:49:07.974046 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:07.974008 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-vtccr" Nov 02 22:49:10.880147 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:10.880110 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:10.880666 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:10.880184 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:10.886054 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:10.886024 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:10.968192 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:10.968161 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:49:37.582462 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:49:37.582419 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 22:49:44.724209 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.724172 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-c989898bf-sb7p9"] Nov 02 22:49:44.777438 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.777403 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:44.791953 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.791924 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c989898bf-sb7p9"] Nov 02 22:49:44.854228 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.854194 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdgk4\" (UniqueName: \"kubernetes.io/projected/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-kube-api-access-xdgk4\") pod \"console-c989898bf-sb7p9\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:44.854228 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.854231 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-tmp\") pod \"console-c989898bf-sb7p9\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:44.854536 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.854251 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-service-ca\") pod \"console-c989898bf-sb7p9\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:44.854536 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.854336 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-console-config\") pod \"console-c989898bf-sb7p9\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:44.854536 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.854383 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-console-serving-cert\") pod \"console-c989898bf-sb7p9\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:44.854536 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.854411 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-console-oauth-config\") pod \"console-c989898bf-sb7p9\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:44.854536 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.854431 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-trusted-ca-bundle\") pod \"console-c989898bf-sb7p9\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:44.854536 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.854520 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-oauth-serving-cert\") pod \"console-c989898bf-sb7p9\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:44.955203 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.955169 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-oauth-serving-cert\") pod \"console-c989898bf-sb7p9\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:44.955401 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.955221 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdgk4\" (UniqueName: \"kubernetes.io/projected/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-kube-api-access-xdgk4\") pod \"console-c989898bf-sb7p9\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:44.955401 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.955348 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-tmp\") pod \"console-c989898bf-sb7p9\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:44.955506 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.955400 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-service-ca\") pod \"console-c989898bf-sb7p9\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:44.955506 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.955441 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-console-config\") pod \"console-c989898bf-sb7p9\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:44.955506 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.955468 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-console-serving-cert\") pod \"console-c989898bf-sb7p9\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:44.955632 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.955599 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-console-oauth-config\") pod \"console-c989898bf-sb7p9\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:44.955682 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.955661 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-trusted-ca-bundle\") pod \"console-c989898bf-sb7p9\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:44.955793 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.955765 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-tmp\") pod \"console-c989898bf-sb7p9\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:44.956064 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.956037 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-oauth-serving-cert\") pod \"console-c989898bf-sb7p9\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:44.956187 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.956170 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-console-config\") pod \"console-c989898bf-sb7p9\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:44.956266 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.956215 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-service-ca\") pod \"console-c989898bf-sb7p9\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:44.956639 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.956615 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-trusted-ca-bundle\") pod \"console-c989898bf-sb7p9\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:44.958235 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.958214 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-console-oauth-config\") pod \"console-c989898bf-sb7p9\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:44.958309 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:44.958254 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-console-serving-cert\") pod \"console-c989898bf-sb7p9\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:45.015913 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:45.015827 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdgk4\" (UniqueName: \"kubernetes.io/projected/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-kube-api-access-xdgk4\") pod \"console-c989898bf-sb7p9\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:45.105378 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:45.105330 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:45.524384 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:45.524350 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c989898bf-sb7p9"] Nov 02 22:49:45.537423 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:49:45.537382 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbdb3215_3611_4ac8_bec8_45b76c7dbbeb.slice/crio-86ae566f420595abfb0e64dd98524c9ad96dfa05595687d9bc8ac3c9a80fe35e WatchSource:0}: Error finding container 86ae566f420595abfb0e64dd98524c9ad96dfa05595687d9bc8ac3c9a80fe35e: Status 404 returned error can't find the container with id 86ae566f420595abfb0e64dd98524c9ad96dfa05595687d9bc8ac3c9a80fe35e Nov 02 22:49:46.079445 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:46.079395 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c989898bf-sb7p9" event={"ID":"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb","Type":"ContainerStarted","Data":"a442c8a68e98160ea1842400cf0f3d015794ad30bcdf75ac097760009cff942e"} Nov 02 22:49:46.079445 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:46.079450 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c989898bf-sb7p9" event={"ID":"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb","Type":"ContainerStarted","Data":"86ae566f420595abfb0e64dd98524c9ad96dfa05595687d9bc8ac3c9a80fe35e"} Nov 02 22:49:46.196074 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:46.195990 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-c989898bf-sb7p9" podStartSLOduration=2.195966148 podStartE2EDuration="2.195966148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:49:46.195898331 +0000 UTC m=+61.548931707" watchObservedRunningTime="2026-11-02 22:49:46.195966148 +0000 UTC m=+61.548999515" Nov 02 22:49:55.105611 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:55.105566 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:55.106249 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:55.105647 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:55.110743 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:55.110714 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:56.116228 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:56.116196 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:49:56.284157 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:49:56.284119 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-759b69bf76-6phhk"] Nov 02 22:50:07.583172 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:50:07.583134 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 22:50:21.305043 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.304979 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-759b69bf76-6phhk" podUID="0b58306c-c5d7-4a3d-a65b-391c79964d11" containerName="console" containerID="cri-o://08f4fcff5b59fcdb09385e292ad93f569fccd4c05a47485958f10e9848938a9e" gracePeriod=15 Nov 02 22:50:21.540885 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.540856 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:50:21.636541 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.636437 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b58306c-c5d7-4a3d-a65b-391c79964d11-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b58306c-c5d7-4a3d-a65b-391c79964d11-console-oauth-config\") pod \"0b58306c-c5d7-4a3d-a65b-391c79964d11\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " Nov 02 22:50:21.636541 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.636511 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-console-config\" (UniqueName: \"kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-console-config\") pod \"0b58306c-c5d7-4a3d-a65b-391c79964d11\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " Nov 02 22:50:21.636770 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.636547 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b58306c-c5d7-4a3d-a65b-391c79964d11-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b58306c-c5d7-4a3d-a65b-391c79964d11-tmp\") pod \"0b58306c-c5d7-4a3d-a65b-391c79964d11\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " Nov 02 22:50:21.636770 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.636564 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b58306c-c5d7-4a3d-a65b-391c79964d11-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b58306c-c5d7-4a3d-a65b-391c79964d11-console-serving-cert\") pod \"0b58306c-c5d7-4a3d-a65b-391c79964d11\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " Nov 02 22:50:21.636770 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.636612 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b58306c-c5d7-4a3d-a65b-391c79964d11-kube-api-access-dxvkk\" (UniqueName: \"kubernetes.io/projected/0b58306c-c5d7-4a3d-a65b-391c79964d11-kube-api-access-dxvkk\") pod \"0b58306c-c5d7-4a3d-a65b-391c79964d11\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " Nov 02 22:50:21.636770 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.636636 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-trusted-ca-bundle\") pod \"0b58306c-c5d7-4a3d-a65b-391c79964d11\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " Nov 02 22:50:21.636770 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.636661 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-oauth-serving-cert\") pod \"0b58306c-c5d7-4a3d-a65b-391c79964d11\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " Nov 02 22:50:21.636770 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.636699 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-service-ca\") pod \"0b58306c-c5d7-4a3d-a65b-391c79964d11\" (UID: \"0b58306c-c5d7-4a3d-a65b-391c79964d11\") " Nov 02 22:50:21.637076 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.636859 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b58306c-c5d7-4a3d-a65b-391c79964d11-tmp" pod "0b58306c-c5d7-4a3d-a65b-391c79964d11" (UID: "0b58306c-c5d7-4a3d-a65b-391c79964d11"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:50:21.637076 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.637032 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-console-config" pod "0b58306c-c5d7-4a3d-a65b-391c79964d11" (UID: "0b58306c-c5d7-4a3d-a65b-391c79964d11"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:50:21.637193 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.637149 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-oauth-serving-cert" pod "0b58306c-c5d7-4a3d-a65b-391c79964d11" (UID: "0b58306c-c5d7-4a3d-a65b-391c79964d11"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:50:21.637193 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.637159 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-trusted-ca-bundle" pod "0b58306c-c5d7-4a3d-a65b-391c79964d11" (UID: "0b58306c-c5d7-4a3d-a65b-391c79964d11"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:50:21.637193 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.637178 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-service-ca" pod "0b58306c-c5d7-4a3d-a65b-391c79964d11" (UID: "0b58306c-c5d7-4a3d-a65b-391c79964d11"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:50:21.638999 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.638979 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b58306c-c5d7-4a3d-a65b-391c79964d11-console-serving-cert" pod "0b58306c-c5d7-4a3d-a65b-391c79964d11" (UID: "0b58306c-c5d7-4a3d-a65b-391c79964d11"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:50:21.639078 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.639022 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b58306c-c5d7-4a3d-a65b-391c79964d11-console-oauth-config" pod "0b58306c-c5d7-4a3d-a65b-391c79964d11" (UID: "0b58306c-c5d7-4a3d-a65b-391c79964d11"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:50:21.639110 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.639072 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b58306c-c5d7-4a3d-a65b-391c79964d11-kube-api-access-dxvkk" pod "0b58306c-c5d7-4a3d-a65b-391c79964d11" (UID: "0b58306c-c5d7-4a3d-a65b-391c79964d11"). InnerVolumeSpecName "kube-api-access-dxvkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:50:21.737754 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.737710 5114 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-console-config\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:50:21.737754 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.737743 5114 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b58306c-c5d7-4a3d-a65b-391c79964d11-tmp\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:50:21.737754 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.737754 5114 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b58306c-c5d7-4a3d-a65b-391c79964d11-console-serving-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:50:21.737997 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.737767 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxvkk\" (UniqueName: \"kubernetes.io/projected/0b58306c-c5d7-4a3d-a65b-391c79964d11-kube-api-access-dxvkk\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:50:21.737997 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.737788 5114 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-trusted-ca-bundle\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:50:21.737997 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.737799 5114 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-oauth-serving-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:50:21.737997 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.737808 5114 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b58306c-c5d7-4a3d-a65b-391c79964d11-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:50:21.737997 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:21.737817 5114 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b58306c-c5d7-4a3d-a65b-391c79964d11-console-oauth-config\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:50:22.198850 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:22.198815 5114 generic.go:356] "Generic (PLEG): container finished" podID="0b58306c-c5d7-4a3d-a65b-391c79964d11" containerID="08f4fcff5b59fcdb09385e292ad93f569fccd4c05a47485958f10e9848938a9e" exitCode=0 Nov 02 22:50:22.199057 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:22.198862 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-759b69bf76-6phhk" event={"ID":"0b58306c-c5d7-4a3d-a65b-391c79964d11","Type":"ContainerDied","Data":"08f4fcff5b59fcdb09385e292ad93f569fccd4c05a47485958f10e9848938a9e"} Nov 02 22:50:22.199057 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:22.198867 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-759b69bf76-6phhk" Nov 02 22:50:22.199057 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:22.198907 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-759b69bf76-6phhk" event={"ID":"0b58306c-c5d7-4a3d-a65b-391c79964d11","Type":"ContainerDied","Data":"ebf8cb549b99f5f51955320a3c8bfcfd7843bf472c2783d271f2c61c569392e0"} Nov 02 22:50:22.199057 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:22.198927 5114 scope.go:122] "RemoveContainer" containerID="08f4fcff5b59fcdb09385e292ad93f569fccd4c05a47485958f10e9848938a9e" Nov 02 22:50:22.207888 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:22.207865 5114 scope.go:122] "RemoveContainer" containerID="08f4fcff5b59fcdb09385e292ad93f569fccd4c05a47485958f10e9848938a9e" Nov 02 22:50:22.208242 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:50:22.208221 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08f4fcff5b59fcdb09385e292ad93f569fccd4c05a47485958f10e9848938a9e\": container with ID starting with 08f4fcff5b59fcdb09385e292ad93f569fccd4c05a47485958f10e9848938a9e not found: ID does not exist" containerID="08f4fcff5b59fcdb09385e292ad93f569fccd4c05a47485958f10e9848938a9e" Nov 02 22:50:22.208299 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:22.208251 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08f4fcff5b59fcdb09385e292ad93f569fccd4c05a47485958f10e9848938a9e"} err="failed to get container status \"08f4fcff5b59fcdb09385e292ad93f569fccd4c05a47485958f10e9848938a9e\": rpc error: code = NotFound desc = could not find container \"08f4fcff5b59fcdb09385e292ad93f569fccd4c05a47485958f10e9848938a9e\": container with ID starting with 08f4fcff5b59fcdb09385e292ad93f569fccd4c05a47485958f10e9848938a9e not found: ID does not exist" Nov 02 22:50:22.361721 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:22.361687 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-759b69bf76-6phhk"] Nov 02 22:50:22.411502 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:22.411448 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-759b69bf76-6phhk"] Nov 02 22:50:22.791401 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:50:22.791353 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b58306c-c5d7-4a3d-a65b-391c79964d11" path="/var/lib/kubelet/pods/0b58306c-c5d7-4a3d-a65b-391c79964d11/volumes" Nov 02 22:50:37.583116 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:50:37.583029 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 22:51:07.582155 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:51:07.582113 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 22:51:08.215382 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.215347 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b577bc47d-h5t9j"] Nov 02 22:51:08.215638 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.215626 5114 cpu_manager.go:414] "containerMap: removing container" podUID="0b58306c-c5d7-4a3d-a65b-391c79964d11" containerName="console" Nov 02 22:51:08.215696 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.215640 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b58306c-c5d7-4a3d-a65b-391c79964d11" containerName="console" Nov 02 22:51:08.215696 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.215685 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b58306c-c5d7-4a3d-a65b-391c79964d11" containerName="console" Nov 02 22:51:08.217390 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.217372 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:08.321773 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.321734 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b577bc47d-h5t9j"] Nov 02 22:51:08.389888 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.389781 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckw6v\" (UniqueName: \"kubernetes.io/projected/98944a14-477b-42ea-bd4b-8e479d23c142-kube-api-access-ckw6v\") pod \"console-6b577bc47d-h5t9j\" (UID: \"98944a14-477b-42ea-bd4b-8e479d23c142\") " pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:08.389888 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.389825 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98944a14-477b-42ea-bd4b-8e479d23c142-oauth-serving-cert\") pod \"console-6b577bc47d-h5t9j\" (UID: \"98944a14-477b-42ea-bd4b-8e479d23c142\") " pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:08.390198 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.389935 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98944a14-477b-42ea-bd4b-8e479d23c142-console-config\") pod \"console-6b577bc47d-h5t9j\" (UID: \"98944a14-477b-42ea-bd4b-8e479d23c142\") " pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:08.390198 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.389981 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98944a14-477b-42ea-bd4b-8e479d23c142-trusted-ca-bundle\") pod \"console-6b577bc47d-h5t9j\" (UID: \"98944a14-477b-42ea-bd4b-8e479d23c142\") " pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:08.390198 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.390062 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98944a14-477b-42ea-bd4b-8e479d23c142-console-serving-cert\") pod \"console-6b577bc47d-h5t9j\" (UID: \"98944a14-477b-42ea-bd4b-8e479d23c142\") " pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:08.390198 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.390086 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98944a14-477b-42ea-bd4b-8e479d23c142-console-oauth-config\") pod \"console-6b577bc47d-h5t9j\" (UID: \"98944a14-477b-42ea-bd4b-8e479d23c142\") " pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:08.390198 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.390114 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98944a14-477b-42ea-bd4b-8e479d23c142-service-ca\") pod \"console-6b577bc47d-h5t9j\" (UID: \"98944a14-477b-42ea-bd4b-8e479d23c142\") " pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:08.390198 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.390134 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98944a14-477b-42ea-bd4b-8e479d23c142-tmp\") pod \"console-6b577bc47d-h5t9j\" (UID: \"98944a14-477b-42ea-bd4b-8e479d23c142\") " pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:08.491174 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.491121 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98944a14-477b-42ea-bd4b-8e479d23c142-service-ca\") pod \"console-6b577bc47d-h5t9j\" (UID: \"98944a14-477b-42ea-bd4b-8e479d23c142\") " pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:08.491174 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.491180 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98944a14-477b-42ea-bd4b-8e479d23c142-tmp\") pod \"console-6b577bc47d-h5t9j\" (UID: \"98944a14-477b-42ea-bd4b-8e479d23c142\") " pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:08.491394 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.491223 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckw6v\" (UniqueName: \"kubernetes.io/projected/98944a14-477b-42ea-bd4b-8e479d23c142-kube-api-access-ckw6v\") pod \"console-6b577bc47d-h5t9j\" (UID: \"98944a14-477b-42ea-bd4b-8e479d23c142\") " pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:08.491394 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.491250 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98944a14-477b-42ea-bd4b-8e479d23c142-oauth-serving-cert\") pod \"console-6b577bc47d-h5t9j\" (UID: \"98944a14-477b-42ea-bd4b-8e479d23c142\") " pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:08.491394 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.491304 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98944a14-477b-42ea-bd4b-8e479d23c142-console-config\") pod \"console-6b577bc47d-h5t9j\" (UID: \"98944a14-477b-42ea-bd4b-8e479d23c142\") " pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:08.491394 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.491332 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98944a14-477b-42ea-bd4b-8e479d23c142-trusted-ca-bundle\") pod \"console-6b577bc47d-h5t9j\" (UID: \"98944a14-477b-42ea-bd4b-8e479d23c142\") " pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:08.491394 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.491368 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98944a14-477b-42ea-bd4b-8e479d23c142-console-serving-cert\") pod \"console-6b577bc47d-h5t9j\" (UID: \"98944a14-477b-42ea-bd4b-8e479d23c142\") " pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:08.491394 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.491393 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98944a14-477b-42ea-bd4b-8e479d23c142-console-oauth-config\") pod \"console-6b577bc47d-h5t9j\" (UID: \"98944a14-477b-42ea-bd4b-8e479d23c142\") " pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:08.491722 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.491689 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98944a14-477b-42ea-bd4b-8e479d23c142-tmp\") pod \"console-6b577bc47d-h5t9j\" (UID: \"98944a14-477b-42ea-bd4b-8e479d23c142\") " pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:08.492038 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.492010 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98944a14-477b-42ea-bd4b-8e479d23c142-service-ca\") pod \"console-6b577bc47d-h5t9j\" (UID: \"98944a14-477b-42ea-bd4b-8e479d23c142\") " pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:08.492147 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.492063 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98944a14-477b-42ea-bd4b-8e479d23c142-oauth-serving-cert\") pod \"console-6b577bc47d-h5t9j\" (UID: \"98944a14-477b-42ea-bd4b-8e479d23c142\") " pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:08.492147 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.492070 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98944a14-477b-42ea-bd4b-8e479d23c142-console-config\") pod \"console-6b577bc47d-h5t9j\" (UID: \"98944a14-477b-42ea-bd4b-8e479d23c142\") " pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:08.492422 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.492271 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98944a14-477b-42ea-bd4b-8e479d23c142-trusted-ca-bundle\") pod \"console-6b577bc47d-h5t9j\" (UID: \"98944a14-477b-42ea-bd4b-8e479d23c142\") " pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:08.494067 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.494038 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98944a14-477b-42ea-bd4b-8e479d23c142-console-oauth-config\") pod \"console-6b577bc47d-h5t9j\" (UID: \"98944a14-477b-42ea-bd4b-8e479d23c142\") " pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:08.494168 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.494130 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98944a14-477b-42ea-bd4b-8e479d23c142-console-serving-cert\") pod \"console-6b577bc47d-h5t9j\" (UID: \"98944a14-477b-42ea-bd4b-8e479d23c142\") " pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:08.552030 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.551994 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckw6v\" (UniqueName: \"kubernetes.io/projected/98944a14-477b-42ea-bd4b-8e479d23c142-kube-api-access-ckw6v\") pod \"console-6b577bc47d-h5t9j\" (UID: \"98944a14-477b-42ea-bd4b-8e479d23c142\") " pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:08.827617 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:08.827581 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:09.247560 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:09.247466 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b577bc47d-h5t9j"] Nov 02 22:51:09.251054 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:51:09.251026 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98944a14_477b_42ea_bd4b_8e479d23c142.slice/crio-9de256b8737f598c1835321e84287863831f45a71b293840549e3b354533d3fd WatchSource:0}: Error finding container 9de256b8737f598c1835321e84287863831f45a71b293840549e3b354533d3fd: Status 404 returned error can't find the container with id 9de256b8737f598c1835321e84287863831f45a71b293840549e3b354533d3fd Nov 02 22:51:09.356589 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:09.356547 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b577bc47d-h5t9j" event={"ID":"98944a14-477b-42ea-bd4b-8e479d23c142","Type":"ContainerStarted","Data":"b3baafe922f781f8b8b68f45ddb51c8e78d086ccd511d8eb2cce8624fccb0fe5"} Nov 02 22:51:09.356780 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:09.356606 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b577bc47d-h5t9j" event={"ID":"98944a14-477b-42ea-bd4b-8e479d23c142","Type":"ContainerStarted","Data":"9de256b8737f598c1835321e84287863831f45a71b293840549e3b354533d3fd"} Nov 02 22:51:09.474163 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:09.474100 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6b577bc47d-h5t9j" podStartSLOduration=1.474080825 podStartE2EDuration="1.474080825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:51:09.47360193 +0000 UTC m=+144.826635306" watchObservedRunningTime="2026-11-02 22:51:09.474080825 +0000 UTC m=+144.827114201" Nov 02 22:51:18.827887 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:18.827838 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:18.828336 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:18.827911 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:18.832827 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:18.832805 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:19.392726 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:19.392694 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b577bc47d-h5t9j" Nov 02 22:51:19.538987 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:19.538951 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c989898bf-sb7p9"] Nov 02 22:51:37.582633 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:51:37.582591 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 22:51:44.559089 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.559017 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-c989898bf-sb7p9" podUID="fbdb3215-3611-4ac8-bec8-45b76c7dbbeb" containerName="console" containerID="cri-o://a442c8a68e98160ea1842400cf0f3d015794ad30bcdf75ac097760009cff942e" gracePeriod=15 Nov 02 22:51:44.809710 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.809642 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:51:44.863899 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.863865 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-trusted-ca-bundle\") pod \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " Nov 02 22:51:44.864113 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.863923 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-oauth-serving-cert\") pod \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " Nov 02 22:51:44.864113 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.863960 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-kube-api-access-xdgk4\" (UniqueName: \"kubernetes.io/projected/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-kube-api-access-xdgk4\") pod \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " Nov 02 22:51:44.864113 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.863981 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-tmp\") pod \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " Nov 02 22:51:44.864113 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.864030 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-console-oauth-config\") pod \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " Nov 02 22:51:44.864113 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.864049 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-console-config\" (UniqueName: \"kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-console-config\") pod \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " Nov 02 22:51:44.864113 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.864088 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-service-ca\") pod \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " Nov 02 22:51:44.864429 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.864272 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-tmp" pod "fbdb3215-3611-4ac8-bec8-45b76c7dbbeb" (UID: "fbdb3215-3611-4ac8-bec8-45b76c7dbbeb"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:51:44.864429 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.864343 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-trusted-ca-bundle" pod "fbdb3215-3611-4ac8-bec8-45b76c7dbbeb" (UID: "fbdb3215-3611-4ac8-bec8-45b76c7dbbeb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:51:44.864429 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.864364 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-console-serving-cert\") pod \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\" (UID: \"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb\") " Nov 02 22:51:44.864607 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.864505 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-oauth-serving-cert" pod "fbdb3215-3611-4ac8-bec8-45b76c7dbbeb" (UID: "fbdb3215-3611-4ac8-bec8-45b76c7dbbeb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:51:44.864607 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.864576 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-console-config" pod "fbdb3215-3611-4ac8-bec8-45b76c7dbbeb" (UID: "fbdb3215-3611-4ac8-bec8-45b76c7dbbeb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:51:44.864750 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.864728 5114 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-console-config\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:51:44.864814 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.864753 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-service-ca" pod "fbdb3215-3611-4ac8-bec8-45b76c7dbbeb" (UID: "fbdb3215-3611-4ac8-bec8-45b76c7dbbeb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:51:44.864814 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.864758 5114 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-trusted-ca-bundle\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:51:44.864814 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.864781 5114 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-oauth-serving-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:51:44.864814 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.864791 5114 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-tmp\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:51:44.866531 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.866471 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-kube-api-access-xdgk4" pod "fbdb3215-3611-4ac8-bec8-45b76c7dbbeb" (UID: "fbdb3215-3611-4ac8-bec8-45b76c7dbbeb"). InnerVolumeSpecName "kube-api-access-xdgk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:51:44.866705 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.866686 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-console-oauth-config" pod "fbdb3215-3611-4ac8-bec8-45b76c7dbbeb" (UID: "fbdb3215-3611-4ac8-bec8-45b76c7dbbeb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:51:44.866947 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.866920 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-console-serving-cert" pod "fbdb3215-3611-4ac8-bec8-45b76c7dbbeb" (UID: "fbdb3215-3611-4ac8-bec8-45b76c7dbbeb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:51:44.965682 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.965641 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdgk4\" (UniqueName: \"kubernetes.io/projected/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-kube-api-access-xdgk4\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:51:44.965682 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.965676 5114 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-console-oauth-config\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:51:44.965682 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.965689 5114 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:51:44.965932 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:44.965697 5114 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb-console-serving-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:51:45.475359 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:45.475322 5114 generic.go:356] "Generic (PLEG): container finished" podID="fbdb3215-3611-4ac8-bec8-45b76c7dbbeb" containerID="a442c8a68e98160ea1842400cf0f3d015794ad30bcdf75ac097760009cff942e" exitCode=0 Nov 02 22:51:45.475566 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:45.475365 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c989898bf-sb7p9" event={"ID":"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb","Type":"ContainerDied","Data":"a442c8a68e98160ea1842400cf0f3d015794ad30bcdf75ac097760009cff942e"} Nov 02 22:51:45.475566 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:45.475388 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c989898bf-sb7p9" Nov 02 22:51:45.475566 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:45.475432 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c989898bf-sb7p9" event={"ID":"fbdb3215-3611-4ac8-bec8-45b76c7dbbeb","Type":"ContainerDied","Data":"86ae566f420595abfb0e64dd98524c9ad96dfa05595687d9bc8ac3c9a80fe35e"} Nov 02 22:51:45.475566 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:45.475452 5114 scope.go:122] "RemoveContainer" containerID="a442c8a68e98160ea1842400cf0f3d015794ad30bcdf75ac097760009cff942e" Nov 02 22:51:45.483905 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:45.483885 5114 scope.go:122] "RemoveContainer" containerID="a442c8a68e98160ea1842400cf0f3d015794ad30bcdf75ac097760009cff942e" Nov 02 22:51:45.484223 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:51:45.484201 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a442c8a68e98160ea1842400cf0f3d015794ad30bcdf75ac097760009cff942e\": container with ID starting with a442c8a68e98160ea1842400cf0f3d015794ad30bcdf75ac097760009cff942e not found: ID does not exist" containerID="a442c8a68e98160ea1842400cf0f3d015794ad30bcdf75ac097760009cff942e" Nov 02 22:51:45.484276 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:45.484235 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a442c8a68e98160ea1842400cf0f3d015794ad30bcdf75ac097760009cff942e"} err="failed to get container status \"a442c8a68e98160ea1842400cf0f3d015794ad30bcdf75ac097760009cff942e\": rpc error: code = NotFound desc = could not find container \"a442c8a68e98160ea1842400cf0f3d015794ad30bcdf75ac097760009cff942e\": container with ID starting with a442c8a68e98160ea1842400cf0f3d015794ad30bcdf75ac097760009cff942e not found: ID does not exist" Nov 02 22:51:45.641844 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:45.641806 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c989898bf-sb7p9"] Nov 02 22:51:45.691504 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:45.691454 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c989898bf-sb7p9"] Nov 02 22:51:46.790943 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:51:46.790902 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbdb3215-3611-4ac8-bec8-45b76c7dbbeb" path="/var/lib/kubelet/pods/fbdb3215-3611-4ac8-bec8-45b76c7dbbeb/volumes" Nov 02 22:52:07.582514 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:52:07.582457 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 22:52:37.584307 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:52:37.584263 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 22:52:56.890577 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:56.890542 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k"] Nov 02 22:52:56.891085 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:56.891025 5114 cpu_manager.go:414] "containerMap: removing container" podUID="fbdb3215-3611-4ac8-bec8-45b76c7dbbeb" containerName="console" Nov 02 22:52:56.891085 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:56.891045 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbdb3215-3611-4ac8-bec8-45b76c7dbbeb" containerName="console" Nov 02 22:52:56.891207 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:56.891121 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbdb3215-3611-4ac8-bec8-45b76c7dbbeb" containerName="console" Nov 02 22:52:56.892881 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:56.892857 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k" Nov 02 22:52:56.990510 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:56.990442 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e5264de-446c-436c-b1b3-adaa13e6a68e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k\" (UID: \"0e5264de-446c-436c-b1b3-adaa13e6a68e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k" Nov 02 22:52:56.990686 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:56.990555 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crlc6\" (UniqueName: \"kubernetes.io/projected/0e5264de-446c-436c-b1b3-adaa13e6a68e-kube-api-access-crlc6\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k\" (UID: \"0e5264de-446c-436c-b1b3-adaa13e6a68e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k" Nov 02 22:52:56.990686 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:56.990584 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e5264de-446c-436c-b1b3-adaa13e6a68e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k\" (UID: \"0e5264de-446c-436c-b1b3-adaa13e6a68e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k" Nov 02 22:52:56.995130 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:56.995105 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 22:52:56.995130 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:56.995119 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-nf9z8\"" Nov 02 22:52:56.995289 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:56.995114 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 22:52:56.995784 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:56.995764 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k"] Nov 02 22:52:57.091117 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.091084 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e5264de-446c-436c-b1b3-adaa13e6a68e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k\" (UID: \"0e5264de-446c-436c-b1b3-adaa13e6a68e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k" Nov 02 22:52:57.091311 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.091132 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crlc6\" (UniqueName: \"kubernetes.io/projected/0e5264de-446c-436c-b1b3-adaa13e6a68e-kube-api-access-crlc6\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k\" (UID: \"0e5264de-446c-436c-b1b3-adaa13e6a68e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k" Nov 02 22:52:57.091311 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.091270 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e5264de-446c-436c-b1b3-adaa13e6a68e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k\" (UID: \"0e5264de-446c-436c-b1b3-adaa13e6a68e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k" Nov 02 22:52:57.091518 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.091470 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e5264de-446c-436c-b1b3-adaa13e6a68e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k\" (UID: \"0e5264de-446c-436c-b1b3-adaa13e6a68e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k" Nov 02 22:52:57.091644 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.091625 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e5264de-446c-436c-b1b3-adaa13e6a68e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k\" (UID: \"0e5264de-446c-436c-b1b3-adaa13e6a68e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k" Nov 02 22:52:57.151965 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.151891 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crlc6\" (UniqueName: \"kubernetes.io/projected/0e5264de-446c-436c-b1b3-adaa13e6a68e-kube-api-access-crlc6\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k\" (UID: \"0e5264de-446c-436c-b1b3-adaa13e6a68e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k" Nov 02 22:52:57.204612 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.204582 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k" Nov 02 22:52:57.291276 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.291243 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn"] Nov 02 22:52:57.293220 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.293197 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:52:57.353691 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.353644 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 22:52:57.353938 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.353724 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 22:52:57.353938 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.353767 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 22:52:57.353938 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.353926 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rdqkr\"" Nov 02 22:52:57.359747 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.359719 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 22:52:57.359860 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.359800 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 22:52:57.359860 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.359804 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 22:52:57.371560 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.371527 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn"] Nov 02 22:52:57.393668 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.393261 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e908152-c102-4895-af2a-d63c925cdee5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-cnxtn\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:52:57.393668 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.393326 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e908152-c102-4895-af2a-d63c925cdee5-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-cnxtn\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:52:57.393668 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.393358 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m94k8\" (UniqueName: \"kubernetes.io/projected/0e908152-c102-4895-af2a-d63c925cdee5-kube-api-access-m94k8\") pod \"istiod-openshift-gateway-7876bfcb74-cnxtn\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:52:57.393668 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.393405 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e908152-c102-4895-af2a-d63c925cdee5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-cnxtn\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:52:57.393668 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.393470 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e908152-c102-4895-af2a-d63c925cdee5-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-cnxtn\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:52:57.393668 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.393520 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e908152-c102-4895-af2a-d63c925cdee5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-cnxtn\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:52:57.393668 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.393553 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e908152-c102-4895-af2a-d63c925cdee5-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-cnxtn\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:52:57.494558 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.494436 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e908152-c102-4895-af2a-d63c925cdee5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-cnxtn\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:52:57.494558 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.494507 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e908152-c102-4895-af2a-d63c925cdee5-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-cnxtn\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:52:57.494558 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.494533 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m94k8\" (UniqueName: \"kubernetes.io/projected/0e908152-c102-4895-af2a-d63c925cdee5-kube-api-access-m94k8\") pod \"istiod-openshift-gateway-7876bfcb74-cnxtn\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:52:57.494844 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.494578 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e908152-c102-4895-af2a-d63c925cdee5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-cnxtn\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:52:57.494844 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.494641 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e908152-c102-4895-af2a-d63c925cdee5-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-cnxtn\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:52:57.494844 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.494671 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e908152-c102-4895-af2a-d63c925cdee5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-cnxtn\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:52:57.494844 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.494784 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e908152-c102-4895-af2a-d63c925cdee5-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-cnxtn\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:52:57.495121 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.495085 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e908152-c102-4895-af2a-d63c925cdee5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-cnxtn\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:52:57.497166 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.497137 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e908152-c102-4895-af2a-d63c925cdee5-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-cnxtn\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:52:57.497259 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.497169 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e908152-c102-4895-af2a-d63c925cdee5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-cnxtn\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:52:57.497309 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.497265 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e908152-c102-4895-af2a-d63c925cdee5-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-cnxtn\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:52:57.497344 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.497311 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e908152-c102-4895-af2a-d63c925cdee5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-cnxtn\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:52:57.555305 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.555267 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e908152-c102-4895-af2a-d63c925cdee5-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-cnxtn\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:52:57.555523 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.555503 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m94k8\" (UniqueName: \"kubernetes.io/projected/0e908152-c102-4895-af2a-d63c925cdee5-kube-api-access-m94k8\") pod \"istiod-openshift-gateway-7876bfcb74-cnxtn\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:52:57.581809 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.581775 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k"] Nov 02 22:52:57.585206 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:52:57.585166 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e5264de_446c_436c_b1b3_adaa13e6a68e.slice/crio-7fb3a9d751a8f5b5a8372b4c4bc8d658b79a4938bf8607f99a71604b438f1882 WatchSource:0}: Error finding container 7fb3a9d751a8f5b5a8372b4c4bc8d658b79a4938bf8607f99a71604b438f1882: Status 404 returned error can't find the container with id 7fb3a9d751a8f5b5a8372b4c4bc8d658b79a4938bf8607f99a71604b438f1882 Nov 02 22:52:57.607077 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.607047 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:52:57.719071 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:57.719031 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k" event={"ID":"0e5264de-446c-436c-b1b3-adaa13e6a68e","Type":"ContainerStarted","Data":"7fb3a9d751a8f5b5a8372b4c4bc8d658b79a4938bf8607f99a71604b438f1882"} Nov 02 22:52:58.028372 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:58.028329 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn"] Nov 02 22:52:58.032673 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:52:58.032636 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e908152_c102_4895_af2a_d63c925cdee5.slice/crio-44a93e3f4c5bdd05f79174db10c548d80a2d7cfef2f1c565131c5974f2f71427 WatchSource:0}: Error finding container 44a93e3f4c5bdd05f79174db10c548d80a2d7cfef2f1c565131c5974f2f71427: Status 404 returned error can't find the container with id 44a93e3f4c5bdd05f79174db10c548d80a2d7cfef2f1c565131c5974f2f71427 Nov 02 22:52:58.724220 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:52:58.724185 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" event={"ID":"0e908152-c102-4895-af2a-d63c925cdee5","Type":"ContainerStarted","Data":"44a93e3f4c5bdd05f79174db10c548d80a2d7cfef2f1c565131c5974f2f71427"} Nov 02 22:53:02.683039 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:02.683007 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:53:02.683419 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:02.683081 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:53:03.756103 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:03.756061 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" event={"ID":"0e908152-c102-4895-af2a-d63c925cdee5","Type":"ContainerStarted","Data":"8ecf369ee7c5864e4d15f0b5ae2eb2d870c81e648612858ed11175d9d38a66d1"} Nov 02 22:53:03.756606 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:03.756234 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:53:03.758382 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:03.758348 5114 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-cnxtn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:53:03.758592 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:03.758410 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" podUID="0e908152-c102-4895-af2a-d63c925cdee5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:53:03.963732 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:03.963661 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" podStartSLOduration=2.316182018 podStartE2EDuration="6.96364014s" totalImagesPullingTime="4.647458122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:03.925372774 +0000 UTC m=+259.278406150" watchObservedRunningTime="2026-11-02 22:53:03.96364014 +0000 UTC m=+259.316673519" Nov 02 22:53:04.760751 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:04.760664 5114 generic.go:356] "Generic (PLEG): container finished" podID="0e5264de-446c-436c-b1b3-adaa13e6a68e" containerID="25a716203089884ab2871339c4617ba472eed09d9f9c01b9672323f0cc153b00" exitCode=0 Nov 02 22:53:04.761233 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:04.760752 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k" event={"ID":"0e5264de-446c-436c-b1b3-adaa13e6a68e","Type":"ContainerDied","Data":"25a716203089884ab2871339c4617ba472eed09d9f9c01b9672323f0cc153b00"} Nov 02 22:53:04.761517 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:04.761471 5114 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-cnxtn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:53:04.761613 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:04.761539 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" podUID="0e908152-c102-4895-af2a-d63c925cdee5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:53:06.770387 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:06.770352 5114 generic.go:356] "Generic (PLEG): container finished" podID="0e5264de-446c-436c-b1b3-adaa13e6a68e" containerID="bd2128f6256d54418156a2a684aeb5699ce39023de482f5b993b6161452ec837" exitCode=0 Nov 02 22:53:06.770829 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:06.770408 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k" event={"ID":"0e5264de-446c-436c-b1b3-adaa13e6a68e","Type":"ContainerDied","Data":"bd2128f6256d54418156a2a684aeb5699ce39023de482f5b993b6161452ec837"} Nov 02 22:53:07.582904 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:53:07.582854 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 22:53:07.761985 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:07.761953 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:53:13.800126 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:13.800075 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k" event={"ID":"0e5264de-446c-436c-b1b3-adaa13e6a68e","Type":"ContainerStarted","Data":"828aa7b9b331c1489609e98ec1d866001033032b37c1a27f86aeef0e2ac59660"} Nov 02 22:53:13.918389 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:13.918329 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k" podStartSLOduration=3.628616397 podStartE2EDuration="17.91831298s" totalImagesPullingTime="14.289696583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:13.917182017 +0000 UTC m=+269.270215404" watchObservedRunningTime="2026-11-02 22:53:13.91831298 +0000 UTC m=+269.271346417" Nov 02 22:53:14.805044 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:14.805002 5114 generic.go:356] "Generic (PLEG): container finished" podID="0e5264de-446c-436c-b1b3-adaa13e6a68e" containerID="828aa7b9b331c1489609e98ec1d866001033032b37c1a27f86aeef0e2ac59660" exitCode=0 Nov 02 22:53:14.805443 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:14.805061 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k" event={"ID":"0e5264de-446c-436c-b1b3-adaa13e6a68e","Type":"ContainerDied","Data":"828aa7b9b331c1489609e98ec1d866001033032b37c1a27f86aeef0e2ac59660"} Nov 02 22:53:15.938679 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:15.938647 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k" Nov 02 22:53:16.050457 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:16.050421 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e5264de-446c-436c-b1b3-adaa13e6a68e-util\" (UniqueName: \"kubernetes.io/empty-dir/0e5264de-446c-436c-b1b3-adaa13e6a68e-util\") pod \"0e5264de-446c-436c-b1b3-adaa13e6a68e\" (UID: \"0e5264de-446c-436c-b1b3-adaa13e6a68e\") " Nov 02 22:53:16.050657 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:16.050498 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e5264de-446c-436c-b1b3-adaa13e6a68e-kube-api-access-crlc6\" (UniqueName: \"kubernetes.io/projected/0e5264de-446c-436c-b1b3-adaa13e6a68e-kube-api-access-crlc6\") pod \"0e5264de-446c-436c-b1b3-adaa13e6a68e\" (UID: \"0e5264de-446c-436c-b1b3-adaa13e6a68e\") " Nov 02 22:53:16.050657 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:16.050571 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e5264de-446c-436c-b1b3-adaa13e6a68e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e5264de-446c-436c-b1b3-adaa13e6a68e-bundle\") pod \"0e5264de-446c-436c-b1b3-adaa13e6a68e\" (UID: \"0e5264de-446c-436c-b1b3-adaa13e6a68e\") " Nov 02 22:53:16.051406 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:16.051375 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5264de-446c-436c-b1b3-adaa13e6a68e-bundle" pod "0e5264de-446c-436c-b1b3-adaa13e6a68e" (UID: "0e5264de-446c-436c-b1b3-adaa13e6a68e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:16.052948 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:16.052920 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e5264de-446c-436c-b1b3-adaa13e6a68e-kube-api-access-crlc6" pod "0e5264de-446c-436c-b1b3-adaa13e6a68e" (UID: "0e5264de-446c-436c-b1b3-adaa13e6a68e"). InnerVolumeSpecName "kube-api-access-crlc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:16.054528 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:16.054503 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5264de-446c-436c-b1b3-adaa13e6a68e-util" pod "0e5264de-446c-436c-b1b3-adaa13e6a68e" (UID: "0e5264de-446c-436c-b1b3-adaa13e6a68e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:16.151361 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:16.151260 5114 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e5264de-446c-436c-b1b3-adaa13e6a68e-bundle\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:16.151361 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:16.151298 5114 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e5264de-446c-436c-b1b3-adaa13e6a68e-util\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:16.151361 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:16.151308 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crlc6\" (UniqueName: \"kubernetes.io/projected/0e5264de-446c-436c-b1b3-adaa13e6a68e-kube-api-access-crlc6\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:53:16.812654 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:16.812628 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k" Nov 02 22:53:16.812813 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:16.812618 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph85k" event={"ID":"0e5264de-446c-436c-b1b3-adaa13e6a68e","Type":"ContainerDied","Data":"7fb3a9d751a8f5b5a8372b4c4bc8d658b79a4938bf8607f99a71604b438f1882"} Nov 02 22:53:16.812813 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:16.812747 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fb3a9d751a8f5b5a8372b4c4bc8d658b79a4938bf8607f99a71604b438f1882" Nov 02 22:53:37.582547 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:53:37.582445 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 22:53:51.848974 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:51.848938 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g"] Nov 02 22:53:51.849404 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:51.849226 5114 cpu_manager.go:414] "containerMap: removing container" podUID="0e5264de-446c-436c-b1b3-adaa13e6a68e" containerName="extract" Nov 02 22:53:51.849404 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:51.849237 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e5264de-446c-436c-b1b3-adaa13e6a68e" containerName="extract" Nov 02 22:53:51.849404 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:51.849250 5114 cpu_manager.go:414] "containerMap: removing container" podUID="0e5264de-446c-436c-b1b3-adaa13e6a68e" containerName="pull" Nov 02 22:53:51.849404 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:51.849255 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e5264de-446c-436c-b1b3-adaa13e6a68e" containerName="pull" Nov 02 22:53:51.849404 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:51.849270 5114 cpu_manager.go:414] "containerMap: removing container" podUID="0e5264de-446c-436c-b1b3-adaa13e6a68e" containerName="util" Nov 02 22:53:51.849404 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:51.849275 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e5264de-446c-436c-b1b3-adaa13e6a68e" containerName="util" Nov 02 22:53:51.849404 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:51.849316 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e5264de-446c-436c-b1b3-adaa13e6a68e" containerName="extract" Nov 02 22:53:51.851149 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:51.851133 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" Nov 02 22:53:51.960976 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:51.960942 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-c5ts9\"" Nov 02 22:53:51.984034 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:51.983998 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g"] Nov 02 22:53:52.001468 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:52.001438 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 22:53:52.001664 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:52.001519 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 22:53:52.001664 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:52.001549 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 22:53:52.001783 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:52.001772 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 22:53:52.001823 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:52.001814 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 22:53:52.038183 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:52.037750 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvj6b\" (UniqueName: \"kubernetes.io/projected/b6bda064-7ede-4905-ab73-0bb536dd1e53-kube-api-access-fvj6b\") pod \"opendatahub-operator-controller-manager-79d467458d-kfm8g\" (UID: \"b6bda064-7ede-4905-ab73-0bb536dd1e53\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" Nov 02 22:53:52.038183 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:52.038069 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6bda064-7ede-4905-ab73-0bb536dd1e53-webhook-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-kfm8g\" (UID: \"b6bda064-7ede-4905-ab73-0bb536dd1e53\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" Nov 02 22:53:52.038183 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:52.038107 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6bda064-7ede-4905-ab73-0bb536dd1e53-tmp\") pod \"opendatahub-operator-controller-manager-79d467458d-kfm8g\" (UID: \"b6bda064-7ede-4905-ab73-0bb536dd1e53\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" Nov 02 22:53:52.038654 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:52.038194 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b6bda064-7ede-4905-ab73-0bb536dd1e53-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-kfm8g\" (UID: \"b6bda064-7ede-4905-ab73-0bb536dd1e53\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" Nov 02 22:53:52.038654 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:52.038261 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b6bda064-7ede-4905-ab73-0bb536dd1e53-manifests\") pod \"opendatahub-operator-controller-manager-79d467458d-kfm8g\" (UID: \"b6bda064-7ede-4905-ab73-0bb536dd1e53\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" Nov 02 22:53:52.038654 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:52.038539 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6bda064-7ede-4905-ab73-0bb536dd1e53-metrics-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-kfm8g\" (UID: \"b6bda064-7ede-4905-ab73-0bb536dd1e53\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" Nov 02 22:53:52.139870 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:52.139760 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6bda064-7ede-4905-ab73-0bb536dd1e53-webhook-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-kfm8g\" (UID: \"b6bda064-7ede-4905-ab73-0bb536dd1e53\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" Nov 02 22:53:52.139870 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:52.139812 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6bda064-7ede-4905-ab73-0bb536dd1e53-tmp\") pod \"opendatahub-operator-controller-manager-79d467458d-kfm8g\" (UID: \"b6bda064-7ede-4905-ab73-0bb536dd1e53\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" Nov 02 22:53:52.139870 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:52.139842 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b6bda064-7ede-4905-ab73-0bb536dd1e53-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-kfm8g\" (UID: \"b6bda064-7ede-4905-ab73-0bb536dd1e53\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" Nov 02 22:53:52.139870 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:52.139868 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b6bda064-7ede-4905-ab73-0bb536dd1e53-manifests\") pod \"opendatahub-operator-controller-manager-79d467458d-kfm8g\" (UID: \"b6bda064-7ede-4905-ab73-0bb536dd1e53\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" Nov 02 22:53:52.140244 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:52.139897 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6bda064-7ede-4905-ab73-0bb536dd1e53-metrics-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-kfm8g\" (UID: \"b6bda064-7ede-4905-ab73-0bb536dd1e53\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" Nov 02 22:53:52.140244 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:52.139942 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvj6b\" (UniqueName: \"kubernetes.io/projected/b6bda064-7ede-4905-ab73-0bb536dd1e53-kube-api-access-fvj6b\") pod \"opendatahub-operator-controller-manager-79d467458d-kfm8g\" (UID: \"b6bda064-7ede-4905-ab73-0bb536dd1e53\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" Nov 02 22:53:52.140458 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:52.140430 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b6bda064-7ede-4905-ab73-0bb536dd1e53-manifests\") pod \"opendatahub-operator-controller-manager-79d467458d-kfm8g\" (UID: \"b6bda064-7ede-4905-ab73-0bb536dd1e53\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" Nov 02 22:53:52.142406 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:52.142381 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6bda064-7ede-4905-ab73-0bb536dd1e53-tmp\") pod \"opendatahub-operator-controller-manager-79d467458d-kfm8g\" (UID: \"b6bda064-7ede-4905-ab73-0bb536dd1e53\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" Nov 02 22:53:52.142758 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:52.142735 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6bda064-7ede-4905-ab73-0bb536dd1e53-metrics-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-kfm8g\" (UID: \"b6bda064-7ede-4905-ab73-0bb536dd1e53\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" Nov 02 22:53:52.142836 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:52.142764 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b6bda064-7ede-4905-ab73-0bb536dd1e53-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-kfm8g\" (UID: \"b6bda064-7ede-4905-ab73-0bb536dd1e53\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" Nov 02 22:53:52.142836 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:52.142764 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6bda064-7ede-4905-ab73-0bb536dd1e53-webhook-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-kfm8g\" (UID: \"b6bda064-7ede-4905-ab73-0bb536dd1e53\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" Nov 02 22:53:52.206632 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:52.206597 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvj6b\" (UniqueName: \"kubernetes.io/projected/b6bda064-7ede-4905-ab73-0bb536dd1e53-kube-api-access-fvj6b\") pod \"opendatahub-operator-controller-manager-79d467458d-kfm8g\" (UID: \"b6bda064-7ede-4905-ab73-0bb536dd1e53\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" Nov 02 22:53:52.463420 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:52.463331 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" Nov 02 22:53:53.025347 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:53.025320 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g"] Nov 02 22:53:53.028015 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:53:53.027989 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6bda064_7ede_4905_ab73_0bb536dd1e53.slice/crio-52335ce57b86810ab7ce9d3e457588654c2247706df8deac4d21c49e09ef5100 WatchSource:0}: Error finding container 52335ce57b86810ab7ce9d3e457588654c2247706df8deac4d21c49e09ef5100: Status 404 returned error can't find the container with id 52335ce57b86810ab7ce9d3e457588654c2247706df8deac4d21c49e09ef5100 Nov 02 22:53:53.029729 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:53.029711 5114 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:53:53.940525 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:53.940471 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" event={"ID":"b6bda064-7ede-4905-ab73-0bb536dd1e53","Type":"ContainerStarted","Data":"52335ce57b86810ab7ce9d3e457588654c2247706df8deac4d21c49e09ef5100"} Nov 02 22:53:56.955415 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:56.955383 5114 generic.go:356] "Generic (PLEG): container finished" podID="b6bda064-7ede-4905-ab73-0bb536dd1e53" containerID="a4004b9638ba9e84c947cc61f384b15b2aac284fd0dcec65edb151f47d606ba7" exitCode=0 Nov 02 22:53:56.955813 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:56.955439 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" event={"ID":"b6bda064-7ede-4905-ab73-0bb536dd1e53","Type":"ContainerDied","Data":"a4004b9638ba9e84c947cc61f384b15b2aac284fd0dcec65edb151f47d606ba7"} Nov 02 22:53:57.961782 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:57.961736 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" event={"ID":"b6bda064-7ede-4905-ab73-0bb536dd1e53","Type":"ContainerStarted","Data":"75650eb8c80641f06dbe49937e6620a9cd15ba10a942111e36240a05c0e9ef12"} Nov 02 22:53:57.962175 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:57.961846 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" Nov 02 22:53:58.116650 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:58.116593 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" podStartSLOduration=2.504497364 podStartE2EDuration="7.116577186s" totalImagesPullingTime="4.200246497s" totalInitContainerRuntime="411.833325ms" isStatefulPod=false podCreationTimestamp="2026-11-02 22:53:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:58.113228096 +0000 UTC m=+313.466261472" watchObservedRunningTime="2026-11-02 22:53:58.116577186 +0000 UTC m=+313.469610560" Nov 02 22:53:59.062378 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:59.062335 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm"] Nov 02 22:53:59.064581 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:59.064559 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm" Nov 02 22:53:59.093320 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:59.093288 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84bd3666-b268-4125-afb2-aa7c39ded1fc-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm\" (UID: \"84bd3666-b268-4125-afb2-aa7c39ded1fc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm" Nov 02 22:53:59.093528 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:59.093327 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvrkk\" (UniqueName: \"kubernetes.io/projected/84bd3666-b268-4125-afb2-aa7c39ded1fc-kube-api-access-pvrkk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm\" (UID: \"84bd3666-b268-4125-afb2-aa7c39ded1fc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm" Nov 02 22:53:59.093528 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:59.093349 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84bd3666-b268-4125-afb2-aa7c39ded1fc-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm\" (UID: \"84bd3666-b268-4125-afb2-aa7c39ded1fc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm" Nov 02 22:53:59.130265 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:59.130230 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-4whqt\"" Nov 02 22:53:59.134278 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:59.134244 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm"] Nov 02 22:53:59.194428 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:59.194385 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84bd3666-b268-4125-afb2-aa7c39ded1fc-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm\" (UID: \"84bd3666-b268-4125-afb2-aa7c39ded1fc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm" Nov 02 22:53:59.194655 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:59.194449 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvrkk\" (UniqueName: \"kubernetes.io/projected/84bd3666-b268-4125-afb2-aa7c39ded1fc-kube-api-access-pvrkk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm\" (UID: \"84bd3666-b268-4125-afb2-aa7c39ded1fc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm" Nov 02 22:53:59.194655 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:59.194502 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84bd3666-b268-4125-afb2-aa7c39ded1fc-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm\" (UID: \"84bd3666-b268-4125-afb2-aa7c39ded1fc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm" Nov 02 22:53:59.194869 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:59.194845 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84bd3666-b268-4125-afb2-aa7c39ded1fc-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm\" (UID: \"84bd3666-b268-4125-afb2-aa7c39ded1fc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm" Nov 02 22:53:59.194998 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:59.194973 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84bd3666-b268-4125-afb2-aa7c39ded1fc-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm\" (UID: \"84bd3666-b268-4125-afb2-aa7c39ded1fc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm" Nov 02 22:53:59.236963 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:59.236933 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 22:53:59.237182 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:59.237141 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 22:53:59.263697 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:59.263652 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvrkk\" (UniqueName: \"kubernetes.io/projected/84bd3666-b268-4125-afb2-aa7c39ded1fc-kube-api-access-pvrkk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm\" (UID: \"84bd3666-b268-4125-afb2-aa7c39ded1fc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm" Nov 02 22:53:59.378178 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:59.378076 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm" Nov 02 22:53:59.754387 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:59.754309 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm"] Nov 02 22:53:59.757877 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:53:59.757847 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84bd3666_b268_4125_afb2_aa7c39ded1fc.slice/crio-41840d03ad8818324e006257db64374931d217056c3075a76b6fe19159c7cd50 WatchSource:0}: Error finding container 41840d03ad8818324e006257db64374931d217056c3075a76b6fe19159c7cd50: Status 404 returned error can't find the container with id 41840d03ad8818324e006257db64374931d217056c3075a76b6fe19159c7cd50 Nov 02 22:53:59.969789 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:59.969754 5114 generic.go:356] "Generic (PLEG): container finished" podID="84bd3666-b268-4125-afb2-aa7c39ded1fc" containerID="2eae18016b0c9447da97867e504b3d38b7d45cc9cd36dc7788790dbe17aee7e2" exitCode=0 Nov 02 22:53:59.969976 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:59.969846 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm" event={"ID":"84bd3666-b268-4125-afb2-aa7c39ded1fc","Type":"ContainerDied","Data":"2eae18016b0c9447da97867e504b3d38b7d45cc9cd36dc7788790dbe17aee7e2"} Nov 02 22:53:59.969976 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:53:59.969915 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm" event={"ID":"84bd3666-b268-4125-afb2-aa7c39ded1fc","Type":"ContainerStarted","Data":"41840d03ad8818324e006257db64374931d217056c3075a76b6fe19159c7cd50"} Nov 02 22:54:04.993934 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:04.993896 5114 generic.go:356] "Generic (PLEG): container finished" podID="84bd3666-b268-4125-afb2-aa7c39ded1fc" containerID="fb42bbdfb7a5c371ab3ad85d75f5de24770367b659f182d5b1c72215b3ea0fb7" exitCode=0 Nov 02 22:54:04.994299 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:04.993960 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm" event={"ID":"84bd3666-b268-4125-afb2-aa7c39ded1fc","Type":"ContainerDied","Data":"fb42bbdfb7a5c371ab3ad85d75f5de24770367b659f182d5b1c72215b3ea0fb7"} Nov 02 22:54:05.999923 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:05.999887 5114 generic.go:356] "Generic (PLEG): container finished" podID="84bd3666-b268-4125-afb2-aa7c39ded1fc" containerID="41687f99a3738d40bc9b9c53bfcc99655ddf4b61d112f593d217a3e4b8923c2f" exitCode=0 Nov 02 22:54:06.000317 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:05.999938 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm" event={"ID":"84bd3666-b268-4125-afb2-aa7c39ded1fc","Type":"ContainerDied","Data":"41687f99a3738d40bc9b9c53bfcc99655ddf4b61d112f593d217a3e4b8923c2f"} Nov 02 22:54:07.125580 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:07.125555 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm" Nov 02 22:54:07.147839 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:07.147807 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84bd3666-b268-4125-afb2-aa7c39ded1fc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/84bd3666-b268-4125-afb2-aa7c39ded1fc-bundle\") pod \"84bd3666-b268-4125-afb2-aa7c39ded1fc\" (UID: \"84bd3666-b268-4125-afb2-aa7c39ded1fc\") " Nov 02 22:54:07.147839 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:07.147854 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84bd3666-b268-4125-afb2-aa7c39ded1fc-kube-api-access-pvrkk\" (UniqueName: \"kubernetes.io/projected/84bd3666-b268-4125-afb2-aa7c39ded1fc-kube-api-access-pvrkk\") pod \"84bd3666-b268-4125-afb2-aa7c39ded1fc\" (UID: \"84bd3666-b268-4125-afb2-aa7c39ded1fc\") " Nov 02 22:54:07.148360 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:07.148334 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84bd3666-b268-4125-afb2-aa7c39ded1fc-bundle" pod "84bd3666-b268-4125-afb2-aa7c39ded1fc" (UID: "84bd3666-b268-4125-afb2-aa7c39ded1fc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:54:07.150214 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:07.150191 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84bd3666-b268-4125-afb2-aa7c39ded1fc-kube-api-access-pvrkk" pod "84bd3666-b268-4125-afb2-aa7c39ded1fc" (UID: "84bd3666-b268-4125-afb2-aa7c39ded1fc"). InnerVolumeSpecName "kube-api-access-pvrkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:54:07.248568 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:07.248531 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84bd3666-b268-4125-afb2-aa7c39ded1fc-util\" (UniqueName: \"kubernetes.io/empty-dir/84bd3666-b268-4125-afb2-aa7c39ded1fc-util\") pod \"84bd3666-b268-4125-afb2-aa7c39ded1fc\" (UID: \"84bd3666-b268-4125-afb2-aa7c39ded1fc\") " Nov 02 22:54:07.248786 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:07.248763 5114 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84bd3666-b268-4125-afb2-aa7c39ded1fc-bundle\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:54:07.248907 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:07.248790 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvrkk\" (UniqueName: \"kubernetes.io/projected/84bd3666-b268-4125-afb2-aa7c39ded1fc-kube-api-access-pvrkk\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:54:07.255471 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:07.255387 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84bd3666-b268-4125-afb2-aa7c39ded1fc-util" pod "84bd3666-b268-4125-afb2-aa7c39ded1fc" (UID: "84bd3666-b268-4125-afb2-aa7c39ded1fc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:54:07.349868 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:07.349830 5114 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84bd3666-b268-4125-afb2-aa7c39ded1fc-util\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:54:07.584606 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:54:07.584561 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 22:54:08.009065 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:08.008971 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm" event={"ID":"84bd3666-b268-4125-afb2-aa7c39ded1fc","Type":"ContainerDied","Data":"41840d03ad8818324e006257db64374931d217056c3075a76b6fe19159c7cd50"} Nov 02 22:54:08.009065 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:08.009028 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41840d03ad8818324e006257db64374931d217056c3075a76b6fe19159c7cd50" Nov 02 22:54:08.009065 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:08.008988 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pmbjm" Nov 02 22:54:08.967354 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:08.967317 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-kfm8g" Nov 02 22:54:37.584539 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:54:37.584463 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 22:54:40.009434 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:40.009399 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-65dbc8865f-58bxj"] Nov 02 22:54:40.009931 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:40.009699 5114 cpu_manager.go:414] "containerMap: removing container" podUID="84bd3666-b268-4125-afb2-aa7c39ded1fc" containerName="util" Nov 02 22:54:40.009931 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:40.009711 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84bd3666-b268-4125-afb2-aa7c39ded1fc" containerName="util" Nov 02 22:54:40.009931 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:40.009721 5114 cpu_manager.go:414] "containerMap: removing container" podUID="84bd3666-b268-4125-afb2-aa7c39ded1fc" containerName="pull" Nov 02 22:54:40.009931 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:40.009726 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84bd3666-b268-4125-afb2-aa7c39ded1fc" containerName="pull" Nov 02 22:54:40.009931 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:40.009741 5114 cpu_manager.go:414] "containerMap: removing container" podUID="84bd3666-b268-4125-afb2-aa7c39ded1fc" containerName="extract" Nov 02 22:54:40.009931 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:40.009745 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84bd3666-b268-4125-afb2-aa7c39ded1fc" containerName="extract" Nov 02 22:54:40.009931 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:40.009800 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="84bd3666-b268-4125-afb2-aa7c39ded1fc" containerName="extract" Nov 02 22:54:40.015038 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:40.015008 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65dbc8865f-58bxj" Nov 02 22:54:40.105823 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:40.105785 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65dbc8865f-58bxj"] Nov 02 22:54:40.119816 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:40.119787 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 22:54:40.119992 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:40.119841 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 22:54:40.119992 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:40.119914 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jpp79\"" Nov 02 22:54:40.122204 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:40.122182 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247dce35-9881-472f-b000-686c2a0a5dfa-tls-certs\") pod \"kube-auth-proxy-65dbc8865f-58bxj\" (UID: \"247dce35-9881-472f-b000-686c2a0a5dfa\") " pod="openshift-ingress/kube-auth-proxy-65dbc8865f-58bxj" Nov 02 22:54:40.122316 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:40.122218 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/247dce35-9881-472f-b000-686c2a0a5dfa-tmp\") pod \"kube-auth-proxy-65dbc8865f-58bxj\" (UID: \"247dce35-9881-472f-b000-686c2a0a5dfa\") " pod="openshift-ingress/kube-auth-proxy-65dbc8865f-58bxj" Nov 02 22:54:40.122316 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:40.122246 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q265n\" (UniqueName: \"kubernetes.io/projected/247dce35-9881-472f-b000-686c2a0a5dfa-kube-api-access-q265n\") pod \"kube-auth-proxy-65dbc8865f-58bxj\" (UID: \"247dce35-9881-472f-b000-686c2a0a5dfa\") " pod="openshift-ingress/kube-auth-proxy-65dbc8865f-58bxj" Nov 02 22:54:40.223062 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:40.223019 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247dce35-9881-472f-b000-686c2a0a5dfa-tls-certs\") pod \"kube-auth-proxy-65dbc8865f-58bxj\" (UID: \"247dce35-9881-472f-b000-686c2a0a5dfa\") " pod="openshift-ingress/kube-auth-proxy-65dbc8865f-58bxj" Nov 02 22:54:40.223062 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:40.223069 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/247dce35-9881-472f-b000-686c2a0a5dfa-tmp\") pod \"kube-auth-proxy-65dbc8865f-58bxj\" (UID: \"247dce35-9881-472f-b000-686c2a0a5dfa\") " pod="openshift-ingress/kube-auth-proxy-65dbc8865f-58bxj" Nov 02 22:54:40.223303 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:40.223091 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q265n\" (UniqueName: \"kubernetes.io/projected/247dce35-9881-472f-b000-686c2a0a5dfa-kube-api-access-q265n\") pod \"kube-auth-proxy-65dbc8865f-58bxj\" (UID: \"247dce35-9881-472f-b000-686c2a0a5dfa\") " pod="openshift-ingress/kube-auth-proxy-65dbc8865f-58bxj" Nov 02 22:54:40.225576 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:40.225547 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/247dce35-9881-472f-b000-686c2a0a5dfa-tmp\") pod \"kube-auth-proxy-65dbc8865f-58bxj\" (UID: \"247dce35-9881-472f-b000-686c2a0a5dfa\") " pod="openshift-ingress/kube-auth-proxy-65dbc8865f-58bxj" Nov 02 22:54:40.225853 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:40.225831 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247dce35-9881-472f-b000-686c2a0a5dfa-tls-certs\") pod \"kube-auth-proxy-65dbc8865f-58bxj\" (UID: \"247dce35-9881-472f-b000-686c2a0a5dfa\") " pod="openshift-ingress/kube-auth-proxy-65dbc8865f-58bxj" Nov 02 22:54:40.284624 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:40.284596 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q265n\" (UniqueName: \"kubernetes.io/projected/247dce35-9881-472f-b000-686c2a0a5dfa-kube-api-access-q265n\") pod \"kube-auth-proxy-65dbc8865f-58bxj\" (UID: \"247dce35-9881-472f-b000-686c2a0a5dfa\") " pod="openshift-ingress/kube-auth-proxy-65dbc8865f-58bxj" Nov 02 22:54:40.325686 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:40.325641 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65dbc8865f-58bxj" Nov 02 22:54:40.696873 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:40.696782 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65dbc8865f-58bxj"] Nov 02 22:54:40.699745 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:54:40.699710 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247dce35_9881_472f_b000_686c2a0a5dfa.slice/crio-3d47361f708e982566d53db081093c8df5c0e248c3ed55ec873aa625723da220 WatchSource:0}: Error finding container 3d47361f708e982566d53db081093c8df5c0e248c3ed55ec873aa625723da220: Status 404 returned error can't find the container with id 3d47361f708e982566d53db081093c8df5c0e248c3ed55ec873aa625723da220 Nov 02 22:54:41.131366 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:41.131329 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65dbc8865f-58bxj" event={"ID":"247dce35-9881-472f-b000-686c2a0a5dfa","Type":"ContainerStarted","Data":"3d47361f708e982566d53db081093c8df5c0e248c3ed55ec873aa625723da220"} Nov 02 22:54:44.146933 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:44.146892 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65dbc8865f-58bxj" event={"ID":"247dce35-9881-472f-b000-686c2a0a5dfa","Type":"ContainerStarted","Data":"13419d90bc331059c9a720d17f43bb20a9f2f70af5c8be5c9ec7cc608c4873c7"} Nov 02 22:54:44.269959 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:44.269905 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-65dbc8865f-58bxj" podStartSLOduration=2.2340382930000002 podStartE2EDuration="5.269887601s" totalImagesPullingTime="3.035849308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:44.267252543 +0000 UTC m=+359.620285917" watchObservedRunningTime="2026-11-02 22:54:44.269887601 +0000 UTC m=+359.622920976" Nov 02 22:54:45.264611 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:45.264575 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-77f99f94cf-4zh75"] Nov 02 22:54:45.267267 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:45.267243 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-77f99f94cf-4zh75" Nov 02 22:54:45.321839 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:45.321800 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 22:54:45.370854 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:45.370821 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85-manager-config\") pod \"jobset-controller-manager-77f99f94cf-4zh75\" (UID: \"53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85\") " pod="openshift-jobset-operator/jobset-controller-manager-77f99f94cf-4zh75" Nov 02 22:54:45.374124 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:45.374080 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85-cert\") pod \"jobset-controller-manager-77f99f94cf-4zh75\" (UID: \"53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85\") " pod="openshift-jobset-operator/jobset-controller-manager-77f99f94cf-4zh75" Nov 02 22:54:45.374353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:45.374204 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhz5g\" (UniqueName: \"kubernetes.io/projected/53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85-kube-api-access-qhz5g\") pod \"jobset-controller-manager-77f99f94cf-4zh75\" (UID: \"53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85\") " pod="openshift-jobset-operator/jobset-controller-manager-77f99f94cf-4zh75" Nov 02 22:54:45.374353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:45.374243 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85-metrics-certs\") pod \"jobset-controller-manager-77f99f94cf-4zh75\" (UID: \"53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85\") " pod="openshift-jobset-operator/jobset-controller-manager-77f99f94cf-4zh75" Nov 02 22:54:45.374353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:45.373863 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 02 22:54:45.374353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:45.373546 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-77f99f94cf-4zh75"] Nov 02 22:54:45.374353 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:45.373964 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:54:45.374676 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:45.373667 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 02 22:54:45.374676 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:45.373785 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-llqhw\"" Nov 02 22:54:45.374676 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:45.373914 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 02 22:54:45.475610 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:45.475573 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85-manager-config\") pod \"jobset-controller-manager-77f99f94cf-4zh75\" (UID: \"53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85\") " pod="openshift-jobset-operator/jobset-controller-manager-77f99f94cf-4zh75" Nov 02 22:54:45.475834 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:45.475617 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85-cert\") pod \"jobset-controller-manager-77f99f94cf-4zh75\" (UID: \"53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85\") " pod="openshift-jobset-operator/jobset-controller-manager-77f99f94cf-4zh75" Nov 02 22:54:45.475834 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:45.475703 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhz5g\" (UniqueName: \"kubernetes.io/projected/53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85-kube-api-access-qhz5g\") pod \"jobset-controller-manager-77f99f94cf-4zh75\" (UID: \"53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85\") " pod="openshift-jobset-operator/jobset-controller-manager-77f99f94cf-4zh75" Nov 02 22:54:45.475834 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:45.475737 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85-metrics-certs\") pod \"jobset-controller-manager-77f99f94cf-4zh75\" (UID: \"53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85\") " pod="openshift-jobset-operator/jobset-controller-manager-77f99f94cf-4zh75" Nov 02 22:54:45.476375 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:45.476351 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85-manager-config\") pod \"jobset-controller-manager-77f99f94cf-4zh75\" (UID: \"53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85\") " pod="openshift-jobset-operator/jobset-controller-manager-77f99f94cf-4zh75" Nov 02 22:54:45.478406 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:45.478382 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85-cert\") pod \"jobset-controller-manager-77f99f94cf-4zh75\" (UID: \"53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85\") " pod="openshift-jobset-operator/jobset-controller-manager-77f99f94cf-4zh75" Nov 02 22:54:45.478527 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:45.478466 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85-metrics-certs\") pod \"jobset-controller-manager-77f99f94cf-4zh75\" (UID: \"53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85\") " pod="openshift-jobset-operator/jobset-controller-manager-77f99f94cf-4zh75" Nov 02 22:54:45.538924 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:45.538889 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhz5g\" (UniqueName: \"kubernetes.io/projected/53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85-kube-api-access-qhz5g\") pod \"jobset-controller-manager-77f99f94cf-4zh75\" (UID: \"53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85\") " pod="openshift-jobset-operator/jobset-controller-manager-77f99f94cf-4zh75" Nov 02 22:54:45.577843 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:45.577801 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-77f99f94cf-4zh75" Nov 02 22:54:45.958636 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:45.958437 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-77f99f94cf-4zh75"] Nov 02 22:54:45.961462 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:54:45.961428 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ebb20c_fe4f_4b41_ad5e_d1c730d6fc85.slice/crio-743b2635889a6878e2992276e638492b2356714a704f3bca2c19e167adfab9bf WatchSource:0}: Error finding container 743b2635889a6878e2992276e638492b2356714a704f3bca2c19e167adfab9bf: Status 404 returned error can't find the container with id 743b2635889a6878e2992276e638492b2356714a704f3bca2c19e167adfab9bf Nov 02 22:54:46.156309 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:46.156271 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-77f99f94cf-4zh75" event={"ID":"53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85","Type":"ContainerStarted","Data":"743b2635889a6878e2992276e638492b2356714a704f3bca2c19e167adfab9bf"} Nov 02 22:54:50.176294 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:50.176251 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-77f99f94cf-4zh75" event={"ID":"53ebb20c-fe4f-4b41-ad5e-d1c730d6fc85","Type":"ContainerStarted","Data":"61117355c6479920e98599c564145b29656621e55b6043a8456f7a40feead3ed"} Nov 02 22:54:50.176781 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:50.176423 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-77f99f94cf-4zh75" Nov 02 22:54:50.310584 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:54:50.310527 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-77f99f94cf-4zh75" podStartSLOduration=1.24965397 podStartE2EDuration="5.310511978s" totalImagesPullingTime="4.060858008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:50.309595482 +0000 UTC m=+365.662628864" watchObservedRunningTime="2026-11-02 22:54:50.310511978 +0000 UTC m=+365.663545353" Nov 02 22:55:01.186878 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:55:01.186838 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-77f99f94cf-4zh75" Nov 02 22:55:07.582899 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:55:07.582854 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 22:55:37.582941 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:55:37.582904 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 22:56:00.651504 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:00.651452 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-bwb4h"] Nov 02 22:56:00.653961 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:00.653940 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-bwb4h" Nov 02 22:56:00.755426 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:00.755385 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 02 22:56:00.755661 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:00.755437 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 02 22:56:00.755661 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:00.755456 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 22:56:00.755661 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:00.755550 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 22:56:00.755661 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:00.755622 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2b2nm\"" Nov 02 22:56:00.756538 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:00.756519 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-bwb4h"] Nov 02 22:56:00.827798 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:00.827757 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wcwb\" (UniqueName: \"kubernetes.io/projected/48efe895-cacd-4880-94d4-3d965679e301-kube-api-access-2wcwb\") pod \"odh-model-controller-769875db56-bwb4h\" (UID: \"48efe895-cacd-4880-94d4-3d965679e301\") " pod="opendatahub/odh-model-controller-769875db56-bwb4h" Nov 02 22:56:00.827798 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:00.827799 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48efe895-cacd-4880-94d4-3d965679e301-cert\") pod \"odh-model-controller-769875db56-bwb4h\" (UID: \"48efe895-cacd-4880-94d4-3d965679e301\") " pod="opendatahub/odh-model-controller-769875db56-bwb4h" Nov 02 22:56:00.828037 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:00.827829 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48efe895-cacd-4880-94d4-3d965679e301-metrics-certs\") pod \"odh-model-controller-769875db56-bwb4h\" (UID: \"48efe895-cacd-4880-94d4-3d965679e301\") " pod="opendatahub/odh-model-controller-769875db56-bwb4h" Nov 02 22:56:00.928934 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:00.928836 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wcwb\" (UniqueName: \"kubernetes.io/projected/48efe895-cacd-4880-94d4-3d965679e301-kube-api-access-2wcwb\") pod \"odh-model-controller-769875db56-bwb4h\" (UID: \"48efe895-cacd-4880-94d4-3d965679e301\") " pod="opendatahub/odh-model-controller-769875db56-bwb4h" Nov 02 22:56:00.928934 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:00.928898 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48efe895-cacd-4880-94d4-3d965679e301-cert\") pod \"odh-model-controller-769875db56-bwb4h\" (UID: \"48efe895-cacd-4880-94d4-3d965679e301\") " pod="opendatahub/odh-model-controller-769875db56-bwb4h" Nov 02 22:56:00.929162 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:00.928945 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48efe895-cacd-4880-94d4-3d965679e301-metrics-certs\") pod \"odh-model-controller-769875db56-bwb4h\" (UID: \"48efe895-cacd-4880-94d4-3d965679e301\") " pod="opendatahub/odh-model-controller-769875db56-bwb4h" Nov 02 22:56:00.931636 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:00.931606 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48efe895-cacd-4880-94d4-3d965679e301-metrics-certs\") pod \"odh-model-controller-769875db56-bwb4h\" (UID: \"48efe895-cacd-4880-94d4-3d965679e301\") " pod="opendatahub/odh-model-controller-769875db56-bwb4h" Nov 02 22:56:00.931763 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:00.931733 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48efe895-cacd-4880-94d4-3d965679e301-cert\") pod \"odh-model-controller-769875db56-bwb4h\" (UID: \"48efe895-cacd-4880-94d4-3d965679e301\") " pod="opendatahub/odh-model-controller-769875db56-bwb4h" Nov 02 22:56:00.990316 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:00.990279 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wcwb\" (UniqueName: \"kubernetes.io/projected/48efe895-cacd-4880-94d4-3d965679e301-kube-api-access-2wcwb\") pod \"odh-model-controller-769875db56-bwb4h\" (UID: \"48efe895-cacd-4880-94d4-3d965679e301\") " pod="opendatahub/odh-model-controller-769875db56-bwb4h" Nov 02 22:56:01.266151 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:01.266052 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-bwb4h" Nov 02 22:56:01.635843 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:01.635810 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-bwb4h"] Nov 02 22:56:01.639046 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:56:01.639017 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48efe895_cacd_4880_94d4_3d965679e301.slice/crio-486349fc28e6c32c22e393ce05aa35449a4c94149e648bcf2ad8554db9227795 WatchSource:0}: Error finding container 486349fc28e6c32c22e393ce05aa35449a4c94149e648bcf2ad8554db9227795: Status 404 returned error can't find the container with id 486349fc28e6c32c22e393ce05aa35449a4c94149e648bcf2ad8554db9227795 Nov 02 22:56:02.475075 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:02.475037 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-bwb4h" event={"ID":"48efe895-cacd-4880-94d4-3d965679e301","Type":"ContainerStarted","Data":"486349fc28e6c32c22e393ce05aa35449a4c94149e648bcf2ad8554db9227795"} Nov 02 22:56:03.993524 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:03.993462 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn"] Nov 02 22:56:03.994003 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:03.993716 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" podUID="0e908152-c102-4895-af2a-d63c925cdee5" containerName="discovery" containerID="cri-o://8ecf369ee7c5864e4d15f0b5ae2eb2d870c81e648612858ed11175d9d38a66d1" gracePeriod=30 Nov 02 22:56:04.254760 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.254693 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:56:04.362713 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.362675 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e908152-c102-4895-af2a-d63c925cdee5-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e908152-c102-4895-af2a-d63c925cdee5-istio-kubeconfig\") pod \"0e908152-c102-4895-af2a-d63c925cdee5\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " Nov 02 22:56:04.362713 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.362725 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e908152-c102-4895-af2a-d63c925cdee5-istio-token\" (UniqueName: \"kubernetes.io/projected/0e908152-c102-4895-af2a-d63c925cdee5-istio-token\") pod \"0e908152-c102-4895-af2a-d63c925cdee5\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " Nov 02 22:56:04.362980 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.362775 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e908152-c102-4895-af2a-d63c925cdee5-cacerts\" (UniqueName: \"kubernetes.io/secret/0e908152-c102-4895-af2a-d63c925cdee5-cacerts\") pod \"0e908152-c102-4895-af2a-d63c925cdee5\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " Nov 02 22:56:04.362980 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.362825 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e908152-c102-4895-af2a-d63c925cdee5-kube-api-access-m94k8\" (UniqueName: \"kubernetes.io/projected/0e908152-c102-4895-af2a-d63c925cdee5-kube-api-access-m94k8\") pod \"0e908152-c102-4895-af2a-d63c925cdee5\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " Nov 02 22:56:04.362980 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.362851 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e908152-c102-4895-af2a-d63c925cdee5-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e908152-c102-4895-af2a-d63c925cdee5-istio-csr-dns-cert\") pod \"0e908152-c102-4895-af2a-d63c925cdee5\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " Nov 02 22:56:04.362980 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.362961 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e908152-c102-4895-af2a-d63c925cdee5-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e908152-c102-4895-af2a-d63c925cdee5-istio-csr-ca-configmap\") pod \"0e908152-c102-4895-af2a-d63c925cdee5\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " Nov 02 22:56:04.363193 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.363001 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e908152-c102-4895-af2a-d63c925cdee5-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e908152-c102-4895-af2a-d63c925cdee5-local-certs\") pod \"0e908152-c102-4895-af2a-d63c925cdee5\" (UID: \"0e908152-c102-4895-af2a-d63c925cdee5\") " Nov 02 22:56:04.364132 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.364075 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e908152-c102-4895-af2a-d63c925cdee5-istio-csr-ca-configmap" pod "0e908152-c102-4895-af2a-d63c925cdee5" (UID: "0e908152-c102-4895-af2a-d63c925cdee5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:56:04.366105 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.366072 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e908152-c102-4895-af2a-d63c925cdee5-cacerts" pod "0e908152-c102-4895-af2a-d63c925cdee5" (UID: "0e908152-c102-4895-af2a-d63c925cdee5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:56:04.366239 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.366170 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e908152-c102-4895-af2a-d63c925cdee5-local-certs" pod "0e908152-c102-4895-af2a-d63c925cdee5" (UID: "0e908152-c102-4895-af2a-d63c925cdee5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:56:04.366239 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.366176 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e908152-c102-4895-af2a-d63c925cdee5-istio-csr-dns-cert" pod "0e908152-c102-4895-af2a-d63c925cdee5" (UID: "0e908152-c102-4895-af2a-d63c925cdee5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:56:04.366360 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.366226 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e908152-c102-4895-af2a-d63c925cdee5-istio-token" pod "0e908152-c102-4895-af2a-d63c925cdee5" (UID: "0e908152-c102-4895-af2a-d63c925cdee5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:56:04.366360 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.366273 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e908152-c102-4895-af2a-d63c925cdee5-istio-kubeconfig" pod "0e908152-c102-4895-af2a-d63c925cdee5" (UID: "0e908152-c102-4895-af2a-d63c925cdee5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:56:04.366832 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.366805 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e908152-c102-4895-af2a-d63c925cdee5-kube-api-access-m94k8" pod "0e908152-c102-4895-af2a-d63c925cdee5" (UID: "0e908152-c102-4895-af2a-d63c925cdee5"). InnerVolumeSpecName "kube-api-access-m94k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:56:04.463970 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.463935 5114 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e908152-c102-4895-af2a-d63c925cdee5-istio-csr-ca-configmap\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:56:04.463970 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.463968 5114 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e908152-c102-4895-af2a-d63c925cdee5-local-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:56:04.463970 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.463979 5114 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e908152-c102-4895-af2a-d63c925cdee5-istio-kubeconfig\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:56:04.464244 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.463989 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e908152-c102-4895-af2a-d63c925cdee5-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:56:04.464244 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.463998 5114 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e908152-c102-4895-af2a-d63c925cdee5-cacerts\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:56:04.464244 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.464007 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m94k8\" (UniqueName: \"kubernetes.io/projected/0e908152-c102-4895-af2a-d63c925cdee5-kube-api-access-m94k8\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:56:04.464244 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.464016 5114 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e908152-c102-4895-af2a-d63c925cdee5-istio-csr-dns-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:56:04.486148 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.486095 5114 generic.go:356] "Generic (PLEG): container finished" podID="0e908152-c102-4895-af2a-d63c925cdee5" containerID="8ecf369ee7c5864e4d15f0b5ae2eb2d870c81e648612858ed11175d9d38a66d1" exitCode=0 Nov 02 22:56:04.486333 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.486220 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" Nov 02 22:56:04.486333 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.486230 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" event={"ID":"0e908152-c102-4895-af2a-d63c925cdee5","Type":"ContainerDied","Data":"8ecf369ee7c5864e4d15f0b5ae2eb2d870c81e648612858ed11175d9d38a66d1"} Nov 02 22:56:04.486333 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.486293 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn" event={"ID":"0e908152-c102-4895-af2a-d63c925cdee5","Type":"ContainerDied","Data":"44a93e3f4c5bdd05f79174db10c548d80a2d7cfef2f1c565131c5974f2f71427"} Nov 02 22:56:04.486333 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.486319 5114 scope.go:122] "RemoveContainer" containerID="8ecf369ee7c5864e4d15f0b5ae2eb2d870c81e648612858ed11175d9d38a66d1" Nov 02 22:56:04.498462 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.498439 5114 scope.go:122] "RemoveContainer" containerID="8ecf369ee7c5864e4d15f0b5ae2eb2d870c81e648612858ed11175d9d38a66d1" Nov 02 22:56:04.498893 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:56:04.498869 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ecf369ee7c5864e4d15f0b5ae2eb2d870c81e648612858ed11175d9d38a66d1\": container with ID starting with 8ecf369ee7c5864e4d15f0b5ae2eb2d870c81e648612858ed11175d9d38a66d1 not found: ID does not exist" containerID="8ecf369ee7c5864e4d15f0b5ae2eb2d870c81e648612858ed11175d9d38a66d1" Nov 02 22:56:04.498980 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.498907 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ecf369ee7c5864e4d15f0b5ae2eb2d870c81e648612858ed11175d9d38a66d1"} err="failed to get container status \"8ecf369ee7c5864e4d15f0b5ae2eb2d870c81e648612858ed11175d9d38a66d1\": rpc error: code = NotFound desc = could not find container \"8ecf369ee7c5864e4d15f0b5ae2eb2d870c81e648612858ed11175d9d38a66d1\": container with ID starting with 8ecf369ee7c5864e4d15f0b5ae2eb2d870c81e648612858ed11175d9d38a66d1 not found: ID does not exist" Nov 02 22:56:04.651808 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.651774 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn"] Nov 02 22:56:04.655447 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.655410 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cnxtn"] Nov 02 22:56:04.797089 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:04.797045 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e908152-c102-4895-af2a-d63c925cdee5" path="/var/lib/kubelet/pods/0e908152-c102-4895-af2a-d63c925cdee5/volumes" Nov 02 22:56:05.492411 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:05.492361 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-bwb4h" event={"ID":"48efe895-cacd-4880-94d4-3d965679e301","Type":"ContainerStarted","Data":"3168c453ceeb2da757b18a5c20241877fb9b15d8923ac0f6f6524b1607b1280b"} Nov 02 22:56:05.492949 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:05.492458 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-bwb4h" Nov 02 22:56:05.609581 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:05.609523 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-bwb4h" podStartSLOduration=2.533716175 podStartE2EDuration="5.609508304s" totalImagesPullingTime="3.075792129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:56:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:56:05.608599168 +0000 UTC m=+440.961632545" watchObservedRunningTime="2026-11-02 22:56:05.609508304 +0000 UTC m=+440.962541676" Nov 02 22:56:05.714428 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:05.714391 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h"] Nov 02 22:56:05.714732 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:05.714718 5114 cpu_manager.go:414] "containerMap: removing container" podUID="0e908152-c102-4895-af2a-d63c925cdee5" containerName="discovery" Nov 02 22:56:05.714788 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:05.714734 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e908152-c102-4895-af2a-d63c925cdee5" containerName="discovery" Nov 02 22:56:05.714827 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:05.714802 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e908152-c102-4895-af2a-d63c925cdee5" containerName="discovery" Nov 02 22:56:05.717869 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:05.717849 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h" Nov 02 22:56:05.777200 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:05.777059 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h"] Nov 02 22:56:05.781225 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:05.780990 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wm4gz\"" Nov 02 22:56:05.781225 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:05.781189 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 22:56:05.781413 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:05.781300 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 22:56:05.831534 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:05.831467 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:56:05.831719 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:05.831578 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:56:05.876220 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:05.876185 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-maas-api-tls\") pod \"maas-api-66fb685bcd-q8p8h\" (UID: \"d0d2dad0-cebc-4792-b09a-0bacf6a4cd22\") " pod="odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h" Nov 02 22:56:05.876220 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:05.876235 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7vb4\" (UniqueName: \"kubernetes.io/projected/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-kube-api-access-v7vb4\") pod \"maas-api-66fb685bcd-q8p8h\" (UID: \"d0d2dad0-cebc-4792-b09a-0bacf6a4cd22\") " pod="odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h" Nov 02 22:56:05.876465 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:05.876256 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-metrics-certs\") pod \"maas-api-66fb685bcd-q8p8h\" (UID: \"d0d2dad0-cebc-4792-b09a-0bacf6a4cd22\") " pod="odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h" Nov 02 22:56:05.876465 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:05.876412 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-openshift-service-ca\") pod \"maas-api-66fb685bcd-q8p8h\" (UID: \"d0d2dad0-cebc-4792-b09a-0bacf6a4cd22\") " pod="odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h" Nov 02 22:56:05.976831 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:05.976793 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-maas-api-tls\") pod \"maas-api-66fb685bcd-q8p8h\" (UID: \"d0d2dad0-cebc-4792-b09a-0bacf6a4cd22\") " pod="odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h" Nov 02 22:56:05.976831 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:05.976842 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7vb4\" (UniqueName: \"kubernetes.io/projected/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-kube-api-access-v7vb4\") pod \"maas-api-66fb685bcd-q8p8h\" (UID: \"d0d2dad0-cebc-4792-b09a-0bacf6a4cd22\") " pod="odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h" Nov 02 22:56:05.977110 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:05.976863 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-metrics-certs\") pod \"maas-api-66fb685bcd-q8p8h\" (UID: \"d0d2dad0-cebc-4792-b09a-0bacf6a4cd22\") " pod="odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h" Nov 02 22:56:05.977110 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:05.976921 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-openshift-service-ca\") pod \"maas-api-66fb685bcd-q8p8h\" (UID: \"d0d2dad0-cebc-4792-b09a-0bacf6a4cd22\") " pod="odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h" Nov 02 22:56:05.979817 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:05.979781 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-maas-api-tls\") pod \"maas-api-66fb685bcd-q8p8h\" (UID: \"d0d2dad0-cebc-4792-b09a-0bacf6a4cd22\") " pod="odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h" Nov 02 22:56:05.979995 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:05.979970 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-metrics-certs\") pod \"maas-api-66fb685bcd-q8p8h\" (UID: \"d0d2dad0-cebc-4792-b09a-0bacf6a4cd22\") " pod="odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h" Nov 02 22:56:05.980435 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:05.980412 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-openshift-service-ca\") pod \"maas-api-66fb685bcd-q8p8h\" (UID: \"d0d2dad0-cebc-4792-b09a-0bacf6a4cd22\") " pod="odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h" Nov 02 22:56:06.038457 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:06.038421 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7vb4\" (UniqueName: \"kubernetes.io/projected/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-kube-api-access-v7vb4\") pod \"maas-api-66fb685bcd-q8p8h\" (UID: \"d0d2dad0-cebc-4792-b09a-0bacf6a4cd22\") " pod="odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h" Nov 02 22:56:06.330892 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:06.330797 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h" Nov 02 22:56:06.704549 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:06.704439 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h"] Nov 02 22:56:06.707619 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:56:06.707586 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d2dad0_cebc_4792_b09a_0bacf6a4cd22.slice/crio-a07f206ae8a4ba1f3ea11dcd21d6e92a1d51e03e304eae3d36d118ccbc08e1d2 WatchSource:0}: Error finding container a07f206ae8a4ba1f3ea11dcd21d6e92a1d51e03e304eae3d36d118ccbc08e1d2: Status 404 returned error can't find the container with id a07f206ae8a4ba1f3ea11dcd21d6e92a1d51e03e304eae3d36d118ccbc08e1d2 Nov 02 22:56:07.502669 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:07.502627 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h" event={"ID":"d0d2dad0-cebc-4792-b09a-0bacf6a4cd22","Type":"ContainerStarted","Data":"a07f206ae8a4ba1f3ea11dcd21d6e92a1d51e03e304eae3d36d118ccbc08e1d2"} Nov 02 22:56:07.583788 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:56:07.583752 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 22:56:13.532946 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:13.532896 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h" event={"ID":"d0d2dad0-cebc-4792-b09a-0bacf6a4cd22","Type":"ContainerStarted","Data":"11c000f27dbb7500cb4d21639c1011b32a49f2317dc085d80c41fd3bf7afa6cb"} Nov 02 22:56:13.533365 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:13.533089 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h" Nov 02 22:56:13.651048 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:13.650984 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h" podStartSLOduration=2.760622215 podStartE2EDuration="8.650971184s" totalImagesPullingTime="5.890348969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:56:13.650942604 +0000 UTC m=+449.003975979" watchObservedRunningTime="2026-11-02 22:56:13.650971184 +0000 UTC m=+449.004004559" Nov 02 22:56:16.499756 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:16.499727 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-bwb4h" Nov 02 22:56:19.543265 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:19.543234 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h" Nov 02 22:56:20.580889 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:20.580852 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-94b5f474-v8lbx"] Nov 02 22:56:20.584138 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:20.584115 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-94b5f474-v8lbx" Nov 02 22:56:20.597720 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:20.597695 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78e093a3-d73a-476f-b2c9-a6f5c452c401-maas-api-tls\") pod \"maas-api-94b5f474-v8lbx\" (UID: \"78e093a3-d73a-476f-b2c9-a6f5c452c401\") " pod="odh-ai-gateway-infra/maas-api-94b5f474-v8lbx" Nov 02 22:56:20.597853 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:20.597741 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78e093a3-d73a-476f-b2c9-a6f5c452c401-metrics-certs\") pod \"maas-api-94b5f474-v8lbx\" (UID: \"78e093a3-d73a-476f-b2c9-a6f5c452c401\") " pod="odh-ai-gateway-infra/maas-api-94b5f474-v8lbx" Nov 02 22:56:20.597853 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:20.597825 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e093a3-d73a-476f-b2c9-a6f5c452c401-openshift-service-ca\") pod \"maas-api-94b5f474-v8lbx\" (UID: \"78e093a3-d73a-476f-b2c9-a6f5c452c401\") " pod="odh-ai-gateway-infra/maas-api-94b5f474-v8lbx" Nov 02 22:56:20.597941 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:20.597867 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5nnc\" (UniqueName: \"kubernetes.io/projected/78e093a3-d73a-476f-b2c9-a6f5c452c401-kube-api-access-d5nnc\") pod \"maas-api-94b5f474-v8lbx\" (UID: \"78e093a3-d73a-476f-b2c9-a6f5c452c401\") " pod="odh-ai-gateway-infra/maas-api-94b5f474-v8lbx" Nov 02 22:56:20.686683 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:20.686644 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-94b5f474-v8lbx"] Nov 02 22:56:20.699263 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:20.699147 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5nnc\" (UniqueName: \"kubernetes.io/projected/78e093a3-d73a-476f-b2c9-a6f5c452c401-kube-api-access-d5nnc\") pod \"maas-api-94b5f474-v8lbx\" (UID: \"78e093a3-d73a-476f-b2c9-a6f5c452c401\") " pod="odh-ai-gateway-infra/maas-api-94b5f474-v8lbx" Nov 02 22:56:20.699551 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:20.699519 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78e093a3-d73a-476f-b2c9-a6f5c452c401-maas-api-tls\") pod \"maas-api-94b5f474-v8lbx\" (UID: \"78e093a3-d73a-476f-b2c9-a6f5c452c401\") " pod="odh-ai-gateway-infra/maas-api-94b5f474-v8lbx" Nov 02 22:56:20.699703 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:20.699594 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78e093a3-d73a-476f-b2c9-a6f5c452c401-metrics-certs\") pod \"maas-api-94b5f474-v8lbx\" (UID: \"78e093a3-d73a-476f-b2c9-a6f5c452c401\") " pod="odh-ai-gateway-infra/maas-api-94b5f474-v8lbx" Nov 02 22:56:20.699703 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:20.699651 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e093a3-d73a-476f-b2c9-a6f5c452c401-openshift-service-ca\") pod \"maas-api-94b5f474-v8lbx\" (UID: \"78e093a3-d73a-476f-b2c9-a6f5c452c401\") " pod="odh-ai-gateway-infra/maas-api-94b5f474-v8lbx" Nov 02 22:56:20.700585 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:20.700561 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e093a3-d73a-476f-b2c9-a6f5c452c401-openshift-service-ca\") pod \"maas-api-94b5f474-v8lbx\" (UID: \"78e093a3-d73a-476f-b2c9-a6f5c452c401\") " pod="odh-ai-gateway-infra/maas-api-94b5f474-v8lbx" Nov 02 22:56:20.702183 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:20.702162 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78e093a3-d73a-476f-b2c9-a6f5c452c401-maas-api-tls\") pod \"maas-api-94b5f474-v8lbx\" (UID: \"78e093a3-d73a-476f-b2c9-a6f5c452c401\") " pod="odh-ai-gateway-infra/maas-api-94b5f474-v8lbx" Nov 02 22:56:20.702268 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:20.702197 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78e093a3-d73a-476f-b2c9-a6f5c452c401-metrics-certs\") pod \"maas-api-94b5f474-v8lbx\" (UID: \"78e093a3-d73a-476f-b2c9-a6f5c452c401\") " pod="odh-ai-gateway-infra/maas-api-94b5f474-v8lbx" Nov 02 22:56:20.763612 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:20.763576 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5nnc\" (UniqueName: \"kubernetes.io/projected/78e093a3-d73a-476f-b2c9-a6f5c452c401-kube-api-access-d5nnc\") pod \"maas-api-94b5f474-v8lbx\" (UID: \"78e093a3-d73a-476f-b2c9-a6f5c452c401\") " pod="odh-ai-gateway-infra/maas-api-94b5f474-v8lbx" Nov 02 22:56:20.900769 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:20.900666 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-94b5f474-v8lbx" Nov 02 22:56:21.274536 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:21.274432 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-94b5f474-v8lbx"] Nov 02 22:56:21.277475 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:56:21.277443 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e093a3_d73a_476f_b2c9_a6f5c452c401.slice/crio-54c2298a30e25b1f4ffb5841fa7cbb27ce53c3e23b50eb094cf6b754f6a881c2 WatchSource:0}: Error finding container 54c2298a30e25b1f4ffb5841fa7cbb27ce53c3e23b50eb094cf6b754f6a881c2: Status 404 returned error can't find the container with id 54c2298a30e25b1f4ffb5841fa7cbb27ce53c3e23b50eb094cf6b754f6a881c2 Nov 02 22:56:21.567387 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:21.567353 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-94b5f474-v8lbx" event={"ID":"78e093a3-d73a-476f-b2c9-a6f5c452c401","Type":"ContainerStarted","Data":"54c2298a30e25b1f4ffb5841fa7cbb27ce53c3e23b50eb094cf6b754f6a881c2"} Nov 02 22:56:22.573274 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:22.573235 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-94b5f474-v8lbx" event={"ID":"78e093a3-d73a-476f-b2c9-a6f5c452c401","Type":"ContainerStarted","Data":"f22e7a66254ea04552d5788635b6df74ef1cc2d5a858f23dd4e274224eccf1ff"} Nov 02 22:56:22.573740 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:22.573391 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-94b5f474-v8lbx" Nov 02 22:56:22.690698 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:22.690632 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-94b5f474-v8lbx" podStartSLOduration=1.835347024 podStartE2EDuration="2.690615033s" totalImagesPullingTime="855.268009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:56:22.689724973 +0000 UTC m=+458.042758372" watchObservedRunningTime="2026-11-02 22:56:22.690615033 +0000 UTC m=+458.043648407" Nov 02 22:56:28.583939 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:28.583858 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-94b5f474-v8lbx" Nov 02 22:56:28.751217 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:28.751183 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h"] Nov 02 22:56:28.751470 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:28.751442 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h" podUID="d0d2dad0-cebc-4792-b09a-0bacf6a4cd22" containerName="maas-api" containerID="cri-o://11c000f27dbb7500cb4d21639c1011b32a49f2317dc085d80c41fd3bf7afa6cb" gracePeriod=30 Nov 02 22:56:29.008284 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:29.008244 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h" Nov 02 22:56:29.072674 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:29.072639 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-openshift-service-ca\") pod \"d0d2dad0-cebc-4792-b09a-0bacf6a4cd22\" (UID: \"d0d2dad0-cebc-4792-b09a-0bacf6a4cd22\") " Nov 02 22:56:29.072884 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:29.072726 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-kube-api-access-v7vb4\" (UniqueName: \"kubernetes.io/projected/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-kube-api-access-v7vb4\") pod \"d0d2dad0-cebc-4792-b09a-0bacf6a4cd22\" (UID: \"d0d2dad0-cebc-4792-b09a-0bacf6a4cd22\") " Nov 02 22:56:29.072884 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:29.072791 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-maas-api-tls\") pod \"d0d2dad0-cebc-4792-b09a-0bacf6a4cd22\" (UID: \"d0d2dad0-cebc-4792-b09a-0bacf6a4cd22\") " Nov 02 22:56:29.072884 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:29.072826 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-metrics-certs\") pod \"d0d2dad0-cebc-4792-b09a-0bacf6a4cd22\" (UID: \"d0d2dad0-cebc-4792-b09a-0bacf6a4cd22\") " Nov 02 22:56:29.073124 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:29.073098 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-openshift-service-ca" pod "d0d2dad0-cebc-4792-b09a-0bacf6a4cd22" (UID: "d0d2dad0-cebc-4792-b09a-0bacf6a4cd22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:56:29.075356 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:29.075323 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-kube-api-access-v7vb4" pod "d0d2dad0-cebc-4792-b09a-0bacf6a4cd22" (UID: "d0d2dad0-cebc-4792-b09a-0bacf6a4cd22"). InnerVolumeSpecName "kube-api-access-v7vb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:56:29.075356 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:29.075340 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-metrics-certs" pod "d0d2dad0-cebc-4792-b09a-0bacf6a4cd22" (UID: "d0d2dad0-cebc-4792-b09a-0bacf6a4cd22"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:56:29.075541 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:29.075440 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-maas-api-tls" pod "d0d2dad0-cebc-4792-b09a-0bacf6a4cd22" (UID: "d0d2dad0-cebc-4792-b09a-0bacf6a4cd22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:56:29.173699 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:29.173617 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-maas-api-tls\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:56:29.173699 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:29.173646 5114 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-metrics-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:56:29.173699 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:29.173656 5114 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-openshift-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:56:29.173699 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:29.173669 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7vb4\" (UniqueName: \"kubernetes.io/projected/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22-kube-api-access-v7vb4\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:56:29.601507 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:29.601442 5114 generic.go:356] "Generic (PLEG): container finished" podID="d0d2dad0-cebc-4792-b09a-0bacf6a4cd22" containerID="11c000f27dbb7500cb4d21639c1011b32a49f2317dc085d80c41fd3bf7afa6cb" exitCode=0 Nov 02 22:56:29.602031 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:29.601524 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h" Nov 02 22:56:29.602031 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:29.601528 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h" event={"ID":"d0d2dad0-cebc-4792-b09a-0bacf6a4cd22","Type":"ContainerDied","Data":"11c000f27dbb7500cb4d21639c1011b32a49f2317dc085d80c41fd3bf7afa6cb"} Nov 02 22:56:29.602031 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:29.601583 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h" event={"ID":"d0d2dad0-cebc-4792-b09a-0bacf6a4cd22","Type":"ContainerDied","Data":"a07f206ae8a4ba1f3ea11dcd21d6e92a1d51e03e304eae3d36d118ccbc08e1d2"} Nov 02 22:56:29.602031 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:29.601606 5114 scope.go:122] "RemoveContainer" containerID="11c000f27dbb7500cb4d21639c1011b32a49f2317dc085d80c41fd3bf7afa6cb" Nov 02 22:56:29.610886 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:29.610864 5114 scope.go:122] "RemoveContainer" containerID="11c000f27dbb7500cb4d21639c1011b32a49f2317dc085d80c41fd3bf7afa6cb" Nov 02 22:56:29.611210 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:56:29.611192 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11c000f27dbb7500cb4d21639c1011b32a49f2317dc085d80c41fd3bf7afa6cb\": container with ID starting with 11c000f27dbb7500cb4d21639c1011b32a49f2317dc085d80c41fd3bf7afa6cb not found: ID does not exist" containerID="11c000f27dbb7500cb4d21639c1011b32a49f2317dc085d80c41fd3bf7afa6cb" Nov 02 22:56:29.611258 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:29.611222 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11c000f27dbb7500cb4d21639c1011b32a49f2317dc085d80c41fd3bf7afa6cb"} err="failed to get container status \"11c000f27dbb7500cb4d21639c1011b32a49f2317dc085d80c41fd3bf7afa6cb\": rpc error: code = NotFound desc = could not find container \"11c000f27dbb7500cb4d21639c1011b32a49f2317dc085d80c41fd3bf7afa6cb\": container with ID starting with 11c000f27dbb7500cb4d21639c1011b32a49f2317dc085d80c41fd3bf7afa6cb not found: ID does not exist" Nov 02 22:56:29.763887 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:29.763849 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h"] Nov 02 22:56:29.813450 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:29.813412 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-66fb685bcd-q8p8h"] Nov 02 22:56:30.791868 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:56:30.791833 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0d2dad0-cebc-4792-b09a-0bacf6a4cd22" path="/var/lib/kubelet/pods/d0d2dad0-cebc-4792-b09a-0bacf6a4cd22/volumes" Nov 02 22:56:37.583405 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:56:37.583367 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 22:57:07.588807 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:57:07.588762 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 22:57:25.354932 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:57:25.354889 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-d6fmn"] Nov 02 22:57:25.355409 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:57:25.355208 5114 cpu_manager.go:414] "containerMap: removing container" podUID="d0d2dad0-cebc-4792-b09a-0bacf6a4cd22" containerName="maas-api" Nov 02 22:57:25.355409 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:57:25.355221 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0d2dad0-cebc-4792-b09a-0bacf6a4cd22" containerName="maas-api" Nov 02 22:57:25.355409 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:57:25.355282 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0d2dad0-cebc-4792-b09a-0bacf6a4cd22" containerName="maas-api" Nov 02 22:57:25.358204 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:57:25.358186 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-d6fmn" Nov 02 22:57:25.421944 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:57:25.421902 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b8dc\" (UniqueName: \"kubernetes.io/projected/289a684e-75c9-4b50-ad59-0ab341e55c61-kube-api-access-7b8dc\") pod \"llmisvc-controller-manager-6466c554d8-d6fmn\" (UID: \"289a684e-75c9-4b50-ad59-0ab341e55c61\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-d6fmn" Nov 02 22:57:25.422125 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:57:25.421956 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/289a684e-75c9-4b50-ad59-0ab341e55c61-cert\") pod \"llmisvc-controller-manager-6466c554d8-d6fmn\" (UID: \"289a684e-75c9-4b50-ad59-0ab341e55c61\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-d6fmn" Nov 02 22:57:25.459888 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:57:25.459857 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-mjzwb\"" Nov 02 22:57:25.460082 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:57:25.459914 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 02 22:57:25.460931 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:57:25.460910 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-d6fmn"] Nov 02 22:57:25.522898 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:57:25.522851 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b8dc\" (UniqueName: \"kubernetes.io/projected/289a684e-75c9-4b50-ad59-0ab341e55c61-kube-api-access-7b8dc\") pod \"llmisvc-controller-manager-6466c554d8-d6fmn\" (UID: \"289a684e-75c9-4b50-ad59-0ab341e55c61\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-d6fmn" Nov 02 22:57:25.523089 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:57:25.522906 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/289a684e-75c9-4b50-ad59-0ab341e55c61-cert\") pod \"llmisvc-controller-manager-6466c554d8-d6fmn\" (UID: \"289a684e-75c9-4b50-ad59-0ab341e55c61\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-d6fmn" Nov 02 22:57:25.525726 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:57:25.525688 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/289a684e-75c9-4b50-ad59-0ab341e55c61-cert\") pod \"llmisvc-controller-manager-6466c554d8-d6fmn\" (UID: \"289a684e-75c9-4b50-ad59-0ab341e55c61\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-d6fmn" Nov 02 22:57:25.584674 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:57:25.584634 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b8dc\" (UniqueName: \"kubernetes.io/projected/289a684e-75c9-4b50-ad59-0ab341e55c61-kube-api-access-7b8dc\") pod \"llmisvc-controller-manager-6466c554d8-d6fmn\" (UID: \"289a684e-75c9-4b50-ad59-0ab341e55c61\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-d6fmn" Nov 02 22:57:25.670793 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:57:25.670669 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-d6fmn" Nov 02 22:57:26.042363 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:57:26.042331 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-d6fmn"] Nov 02 22:57:26.045745 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:57:26.045720 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289a684e_75c9_4b50_ad59_0ab341e55c61.slice/crio-b48de4b1a775ce6a0d1c0ae875cbc239e4298fe055cd69fe3b2ae7a9370d7bcd WatchSource:0}: Error finding container b48de4b1a775ce6a0d1c0ae875cbc239e4298fe055cd69fe3b2ae7a9370d7bcd: Status 404 returned error can't find the container with id b48de4b1a775ce6a0d1c0ae875cbc239e4298fe055cd69fe3b2ae7a9370d7bcd Nov 02 22:57:26.820179 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:57:26.820133 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-d6fmn" event={"ID":"289a684e-75c9-4b50-ad59-0ab341e55c61","Type":"ContainerStarted","Data":"b48de4b1a775ce6a0d1c0ae875cbc239e4298fe055cd69fe3b2ae7a9370d7bcd"} Nov 02 22:57:29.832920 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:57:29.832872 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-d6fmn" event={"ID":"289a684e-75c9-4b50-ad59-0ab341e55c61","Type":"ContainerStarted","Data":"28780b93c93c963083246ae30ddb8056c9e6ba0cce44678dd0d25bb2b4ad1742"} Nov 02 22:57:29.833357 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:57:29.833008 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-d6fmn" Nov 02 22:57:29.949876 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:57:29.949821 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-d6fmn" podStartSLOduration=2.193915966 podStartE2EDuration="4.949804483s" totalImagesPullingTime="2.755888517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:57:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:57:29.948590724 +0000 UTC m=+525.301624119" watchObservedRunningTime="2026-11-02 22:57:29.949804483 +0000 UTC m=+525.302837857" Nov 02 22:57:37.583771 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:57:37.583734 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 22:58:00.839626 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:58:00.839590 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-d6fmn" Nov 02 22:58:07.583046 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:58:07.583008 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 22:58:37.583840 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:58:37.583799 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 22:59:07.583068 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:59:07.583027 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 22:59:08.322229 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.322187 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf"] Nov 02 22:59:08.325505 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.325470 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.377350 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.377291 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-98x6w\"" Nov 02 22:59:08.377550 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.377415 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 22:59:08.428241 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.428210 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf"] Nov 02 22:59:08.489412 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.489370 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.489412 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.489416 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa4c136-742e-4387-b751-87692a034ba2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.489678 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.489439 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.489678 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.489581 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.489678 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.489613 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfqtg\" (UniqueName: \"kubernetes.io/projected/6aa4c136-742e-4387-b751-87692a034ba2-kube-api-access-bfqtg\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.489678 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.489651 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.489678 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.489673 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa4c136-742e-4387-b751-87692a034ba2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.489848 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.489689 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa4c136-742e-4387-b751-87692a034ba2-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.489848 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.489763 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.590817 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.590726 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.590817 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.590780 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.590817 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.590803 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa4c136-742e-4387-b751-87692a034ba2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.591441 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.590830 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.591441 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.590856 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.591441 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.590878 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfqtg\" (UniqueName: \"kubernetes.io/projected/6aa4c136-742e-4387-b751-87692a034ba2-kube-api-access-bfqtg\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.591441 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.590905 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.591441 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.590936 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa4c136-742e-4387-b751-87692a034ba2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.591441 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.590961 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa4c136-742e-4387-b751-87692a034ba2-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.591441 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.591197 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.591441 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.591368 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.591441 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.591404 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.591932 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.591447 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.591932 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.591861 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa4c136-742e-4387-b751-87692a034ba2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.593210 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.593184 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.593634 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.593611 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa4c136-742e-4387-b751-87692a034ba2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.652225 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.652190 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa4c136-742e-4387-b751-87692a034ba2-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.652446 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.652285 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfqtg\" (UniqueName: \"kubernetes.io/projected/6aa4c136-742e-4387-b751-87692a034ba2-kube-api-access-bfqtg\") pod \"maas-default-gateway-openshift-default-694db55c87-jgsxf\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:08.937791 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:08.937703 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:09.364611 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:09.364573 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf"] Nov 02 22:59:09.367838 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:59:09.367807 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa4c136_742e_4387_b751_87692a034ba2.slice/crio-ce0bc5b41e623b5b97ccc0b8f36a145574ca7f2492b81bed9e563c1adff3c20f WatchSource:0}: Error finding container ce0bc5b41e623b5b97ccc0b8f36a145574ca7f2492b81bed9e563c1adff3c20f: Status 404 returned error can't find the container with id ce0bc5b41e623b5b97ccc0b8f36a145574ca7f2492b81bed9e563c1adff3c20f Nov 02 22:59:09.369951 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:09.369933 5114 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:59:10.247499 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:10.247447 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" event={"ID":"6aa4c136-742e-4387-b751-87692a034ba2","Type":"ContainerStarted","Data":"ce0bc5b41e623b5b97ccc0b8f36a145574ca7f2492b81bed9e563c1adff3c20f"} Nov 02 22:59:12.750715 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:12.750679 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:59:12.751108 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:12.750764 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:59:12.751108 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:12.750812 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:59:13.262046 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:13.262001 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" event={"ID":"6aa4c136-742e-4387-b751-87692a034ba2","Type":"ContainerStarted","Data":"41634aae1ce9ce1a01cabfeb30ea5867244fbb8c0e5a004a891285754b1d7cd8"} Nov 02 22:59:13.433090 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:13.432843 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" podStartSLOduration=2.053089158 podStartE2EDuration="5.432821768s" totalImagesPullingTime="3.37973261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:59:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:59:13.432071371 +0000 UTC m=+628.785104751" watchObservedRunningTime="2026-11-02 22:59:13.432821768 +0000 UTC m=+628.785855142" Nov 02 22:59:13.938752 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:13.938704 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:13.940514 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:13.940468 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-jgsxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Nov 02 22:59:13.940676 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:13.940550 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" podUID="6aa4c136-742e-4387-b751-87692a034ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Nov 02 22:59:14.258915 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.258840 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4"] Nov 02 22:59:14.262158 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.262133 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 22:59:14.338810 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.338773 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/832b80ef-5c2c-4791-81c4-b92748d79552-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4\" (UID: \"832b80ef-5c2c-4791-81c4-b92748d79552\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 22:59:14.338810 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.338812 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/832b80ef-5c2c-4791-81c4-b92748d79552-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4\" (UID: \"832b80ef-5c2c-4791-81c4-b92748d79552\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 22:59:14.339025 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.338903 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/832b80ef-5c2c-4791-81c4-b92748d79552-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4\" (UID: \"832b80ef-5c2c-4791-81c4-b92748d79552\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 22:59:14.339025 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.338979 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/832b80ef-5c2c-4791-81c4-b92748d79552-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4\" (UID: \"832b80ef-5c2c-4791-81c4-b92748d79552\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 22:59:14.339247 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.339221 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7nj5\" (UniqueName: \"kubernetes.io/projected/832b80ef-5c2c-4791-81c4-b92748d79552-kube-api-access-n7nj5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4\" (UID: \"832b80ef-5c2c-4791-81c4-b92748d79552\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 22:59:14.339325 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.339274 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/832b80ef-5c2c-4791-81c4-b92748d79552-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4\" (UID: \"832b80ef-5c2c-4791-81c4-b92748d79552\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 22:59:14.381587 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.381552 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tzw2x\"" Nov 02 22:59:14.381587 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.381549 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 22:59:14.381828 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.381628 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 22:59:14.381828 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.381705 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 02 22:59:14.382737 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.382719 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4"] Nov 02 22:59:14.439734 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.439686 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/832b80ef-5c2c-4791-81c4-b92748d79552-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4\" (UID: \"832b80ef-5c2c-4791-81c4-b92748d79552\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 22:59:14.439734 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.439738 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7nj5\" (UniqueName: \"kubernetes.io/projected/832b80ef-5c2c-4791-81c4-b92748d79552-kube-api-access-n7nj5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4\" (UID: \"832b80ef-5c2c-4791-81c4-b92748d79552\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 22:59:14.439993 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.439762 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/832b80ef-5c2c-4791-81c4-b92748d79552-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4\" (UID: \"832b80ef-5c2c-4791-81c4-b92748d79552\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 22:59:14.439993 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.439782 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/832b80ef-5c2c-4791-81c4-b92748d79552-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4\" (UID: \"832b80ef-5c2c-4791-81c4-b92748d79552\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 22:59:14.439993 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.439797 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/832b80ef-5c2c-4791-81c4-b92748d79552-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4\" (UID: \"832b80ef-5c2c-4791-81c4-b92748d79552\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 22:59:14.439993 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.439824 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/832b80ef-5c2c-4791-81c4-b92748d79552-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4\" (UID: \"832b80ef-5c2c-4791-81c4-b92748d79552\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 22:59:14.440233 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.440210 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/832b80ef-5c2c-4791-81c4-b92748d79552-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4\" (UID: \"832b80ef-5c2c-4791-81c4-b92748d79552\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 22:59:14.440313 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.440290 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/832b80ef-5c2c-4791-81c4-b92748d79552-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4\" (UID: \"832b80ef-5c2c-4791-81c4-b92748d79552\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 22:59:14.440374 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.440331 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/832b80ef-5c2c-4791-81c4-b92748d79552-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4\" (UID: \"832b80ef-5c2c-4791-81c4-b92748d79552\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 22:59:14.442138 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.442121 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/832b80ef-5c2c-4791-81c4-b92748d79552-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4\" (UID: \"832b80ef-5c2c-4791-81c4-b92748d79552\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 22:59:14.442538 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.442513 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/832b80ef-5c2c-4791-81c4-b92748d79552-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4\" (UID: \"832b80ef-5c2c-4791-81c4-b92748d79552\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 22:59:14.501323 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.501288 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7nj5\" (UniqueName: \"kubernetes.io/projected/832b80ef-5c2c-4791-81c4-b92748d79552-kube-api-access-n7nj5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4\" (UID: \"832b80ef-5c2c-4791-81c4-b92748d79552\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 22:59:14.581931 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.581897 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 22:59:14.938552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.938439 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-jgsxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Nov 02 22:59:14.938552 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.938530 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" podUID="6aa4c136-742e-4387-b751-87692a034ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Nov 02 22:59:14.977683 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:14.977652 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4"] Nov 02 22:59:14.981558 ip-10-0-135-250 kubenswrapper[5114]: W1102 22:59:14.981529 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod832b80ef_5c2c_4791_81c4_b92748d79552.slice/crio-0b220769ecf7cd81adece2ee3af6c7624b9e86d3cf8710a41a21ef602aa25523 WatchSource:0}: Error finding container 0b220769ecf7cd81adece2ee3af6c7624b9e86d3cf8710a41a21ef602aa25523: Status 404 returned error can't find the container with id 0b220769ecf7cd81adece2ee3af6c7624b9e86d3cf8710a41a21ef602aa25523 Nov 02 22:59:15.271256 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:15.271167 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" event={"ID":"832b80ef-5c2c-4791-81c4-b92748d79552","Type":"ContainerStarted","Data":"0b220769ecf7cd81adece2ee3af6c7624b9e86d3cf8710a41a21ef602aa25523"} Nov 02 22:59:15.938572 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:15.938539 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-jgsxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Nov 02 22:59:15.938788 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:15.938598 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" podUID="6aa4c136-742e-4387-b751-87692a034ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Nov 02 22:59:16.938205 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:16.938162 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-jgsxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Nov 02 22:59:16.938770 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:16.938248 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" podUID="6aa4c136-742e-4387-b751-87692a034ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Nov 02 22:59:17.282001 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:17.281894 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" event={"ID":"832b80ef-5c2c-4791-81c4-b92748d79552","Type":"ContainerStarted","Data":"692292e84dfdba2df5000f341f7cc63ec71d728635221e8eaf7c9756405f64f7"} Nov 02 22:59:17.399923 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:17.399870 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" podStartSLOduration=1.300739017 podStartE2EDuration="3.399854478s" totalImagesPullingTime="2.099115461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:59:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:59:17.398774327 +0000 UTC m=+632.751807703" watchObservedRunningTime="2026-11-02 22:59:17.399854478 +0000 UTC m=+632.752887852" Nov 02 22:59:17.582259 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:17.582221 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 22:59:17.939111 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:17.939015 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-jgsxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Nov 02 22:59:17.939111 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:17.939076 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" podUID="6aa4c136-742e-4387-b751-87692a034ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Nov 02 22:59:18.938969 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:18.938929 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-jgsxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Nov 02 22:59:18.939168 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:18.938998 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" podUID="6aa4c136-742e-4387-b751-87692a034ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Nov 02 22:59:20.556438 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:20.556403 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:20.556926 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:20.556818 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:20.557601 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:20.557582 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 22:59:24.583162 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:24.583071 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 22:59:24.595860 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:24.595829 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 22:59:25.328752 ip-10-0-135-250 kubenswrapper[5114]: I1102 22:59:25.328722 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 22:59:37.583742 ip-10-0-135-250 kubenswrapper[5114]: E1102 22:59:37.583705 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:00:07.584587 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:00:07.584509 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:00:14.997906 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:14.997867 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm"] Nov 02 23:00:15.001474 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.001445 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.053656 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.053617 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5dd28258-openshift-default-dockercfg-ww8hw\"" Nov 02 23:00:15.072601 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.072557 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6xpf\" (UniqueName: \"kubernetes.io/projected/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-kube-api-access-x6xpf\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.072802 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.072695 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-credential-socket\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.072802 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.072737 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-podinfo\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.072802 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.072759 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istiod-ca-cert\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.072934 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.072911 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-data\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.072975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.072956 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-workload-certs\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.073017 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.072995 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-envoy\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.073095 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.073078 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-workload-socket\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.073140 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.073126 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-token\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.104380 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.104335 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm"] Nov 02 23:00:15.174453 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.174414 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-credential-socket\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.174692 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.174460 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-podinfo\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.174692 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.174516 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istiod-ca-cert\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.174692 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.174552 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-data\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.174692 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.174579 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-workload-certs\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.174692 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.174604 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-envoy\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.175013 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.174770 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-workload-socket\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.175013 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.174849 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-token\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.175013 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.174882 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6xpf\" (UniqueName: \"kubernetes.io/projected/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-kube-api-access-x6xpf\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.175013 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.174965 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-credential-socket\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.175202 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.175077 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-data\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.175202 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.175133 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-workload-socket\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.175302 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.175225 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-workload-certs\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.175847 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.175821 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istiod-ca-cert\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.177092 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.177074 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-envoy\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.177233 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.177215 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-podinfo\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.237504 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.237461 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6xpf\" (UniqueName: \"kubernetes.io/projected/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-kube-api-access-x6xpf\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.237504 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.237468 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-token\") pod \"e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.315010 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.314958 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:15.747183 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.747071 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm"] Nov 02 23:00:15.749911 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:00:15.749878 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dad888a_7ae1_481a_8f3d_877d6a1c52bf.slice/crio-b8fba98b7a862265304b987c1719ab2255eace79ab8347063a372ede93633c20 WatchSource:0}: Error finding container b8fba98b7a862265304b987c1719ab2255eace79ab8347063a372ede93633c20: Status 404 returned error can't find the container with id b8fba98b7a862265304b987c1719ab2255eace79ab8347063a372ede93633c20 Nov 02 23:00:15.752019 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.751987 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:00:15.752085 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.752057 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:00:15.752121 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:15.752085 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:00:16.524353 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:16.524323 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" event={"ID":"0dad888a-7ae1-481a-8f3d-877d6a1c52bf","Type":"ContainerStarted","Data":"fc48f2d69d86c43425ed0cc0c6d065bfaf15827b972204f4236b8d6c70c8c5b4"} Nov 02 23:00:16.524775 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:16.524376 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" event={"ID":"0dad888a-7ae1-481a-8f3d-877d6a1c52bf","Type":"ContainerStarted","Data":"b8fba98b7a862265304b987c1719ab2255eace79ab8347063a372ede93633c20"} Nov 02 23:00:16.693962 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:16.693906 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" podStartSLOduration=2.6938901509999997 podStartE2EDuration="2.693890151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:00:16.692114677 +0000 UTC m=+692.045148052" watchObservedRunningTime="2026-11-02 23:00:16.693890151 +0000 UTC m=+692.046923525" Nov 02 23:00:17.316242 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:17.316199 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:17.321665 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:17.321633 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:17.528221 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:17.528189 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:17.529274 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:17.529255 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:19.789909 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:19.789875 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6"] Nov 02 23:00:19.793405 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:19.793383 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:19.846137 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:19.846104 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-708f46a0-openshift-default-dockercfg-jkdgn\"" Nov 02 23:00:19.897559 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:19.897523 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6"] Nov 02 23:00:19.925614 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:19.925578 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cxpz\" (UniqueName: \"kubernetes.io/projected/d8eb89e7-8060-404a-b694-9d7bca211dac-kube-api-access-2cxpz\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:19.925789 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:19.925626 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-workload-certs\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:19.925789 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:19.925646 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8eb89e7-8060-404a-b694-9d7bca211dac-istiod-ca-cert\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:19.925789 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:19.925769 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-data\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:19.925902 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:19.925815 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-envoy\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:19.925902 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:19.925841 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-podinfo\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:19.925902 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:19.925884 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-token\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:19.925989 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:19.925978 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-workload-socket\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:19.926021 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:19.925998 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-credential-socket\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:20.026838 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:20.026795 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-data\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:20.027061 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:20.026859 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-envoy\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:20.027061 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:20.026886 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-podinfo\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:20.027061 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:20.026918 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-token\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:20.027061 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:20.026983 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-workload-socket\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:20.027061 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:20.027011 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-credential-socket\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:20.027061 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:20.027045 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cxpz\" (UniqueName: \"kubernetes.io/projected/d8eb89e7-8060-404a-b694-9d7bca211dac-kube-api-access-2cxpz\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:20.027382 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:20.027091 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-workload-certs\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:20.027382 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:20.027116 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8eb89e7-8060-404a-b694-9d7bca211dac-istiod-ca-cert\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:20.027382 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:20.027282 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-data\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:20.027579 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:20.027402 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-workload-socket\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:20.027579 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:20.027552 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-workload-certs\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:20.027845 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:20.027674 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-credential-socket\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:20.027845 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:20.027824 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8eb89e7-8060-404a-b694-9d7bca211dac-istiod-ca-cert\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:20.029421 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:20.029397 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-envoy\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:20.029705 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:20.029687 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-podinfo\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:20.088415 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:20.088329 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-token\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:20.088719 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:20.088699 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cxpz\" (UniqueName: \"kubernetes.io/projected/d8eb89e7-8060-404a-b694-9d7bca211dac-kube-api-access-2cxpz\") pod \"e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:20.106709 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:20.106673 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:20.534047 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:20.534009 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6"] Nov 02 23:00:20.538616 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:00:20.538580 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8eb89e7_8060_404a_b694_9d7bca211dac.slice/crio-2a61f275a58f79f841ad420d40e34ddd1c8381d7fa5584a3cba6b3f0aca89fed WatchSource:0}: Error finding container 2a61f275a58f79f841ad420d40e34ddd1c8381d7fa5584a3cba6b3f0aca89fed: Status 404 returned error can't find the container with id 2a61f275a58f79f841ad420d40e34ddd1c8381d7fa5584a3cba6b3f0aca89fed Nov 02 23:00:20.541094 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:20.541060 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:00:20.541196 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:20.541145 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:00:20.541196 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:20.541183 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:00:21.549187 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:21.549145 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" event={"ID":"d8eb89e7-8060-404a-b694-9d7bca211dac","Type":"ContainerStarted","Data":"de68994260b9c591da31a9bef0542c4fe61b04beb69013b7ab87337d1a74f906"} Nov 02 23:00:21.549597 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:21.549204 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" event={"ID":"d8eb89e7-8060-404a-b694-9d7bca211dac","Type":"ContainerStarted","Data":"2a61f275a58f79f841ad420d40e34ddd1c8381d7fa5584a3cba6b3f0aca89fed"} Nov 02 23:00:21.768394 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:21.768338 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" podStartSLOduration=2.768323457 podStartE2EDuration="2.768323457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:00:21.718709944 +0000 UTC m=+697.071743320" watchObservedRunningTime="2026-11-02 23:00:21.768323457 +0000 UTC m=+697.121356831" Nov 02 23:00:22.107700 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:22.107659 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:22.112658 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:22.112629 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:22.557282 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:22.557248 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:22.558580 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:22.558555 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:00:28.000346 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:28.000313 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs"] Nov 02 23:00:28.003663 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:28.003635 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" Nov 02 23:00:28.055564 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:28.055534 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5dd28258-serving-cert\"" Nov 02 23:00:28.105199 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:28.105159 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-metrics-certs\") pod \"maas-api-e2e-ait-5dd28258-85867d745c-x8zfs\" (UID: \"1a008dc0-b654-4d0d-8440-fcc3e6f0def6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" Nov 02 23:00:28.105376 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:28.105215 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-maas-api-tls\") pod \"maas-api-e2e-ait-5dd28258-85867d745c-x8zfs\" (UID: \"1a008dc0-b654-4d0d-8440-fcc3e6f0def6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" Nov 02 23:00:28.105376 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:28.105292 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfj9r\" (UniqueName: \"kubernetes.io/projected/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-kube-api-access-nfj9r\") pod \"maas-api-e2e-ait-5dd28258-85867d745c-x8zfs\" (UID: \"1a008dc0-b654-4d0d-8440-fcc3e6f0def6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" Nov 02 23:00:28.105448 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:28.105399 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-openshift-service-ca\") pod \"maas-api-e2e-ait-5dd28258-85867d745c-x8zfs\" (UID: \"1a008dc0-b654-4d0d-8440-fcc3e6f0def6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" Nov 02 23:00:28.110209 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:28.110183 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs"] Nov 02 23:00:28.206137 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:28.206093 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-metrics-certs\") pod \"maas-api-e2e-ait-5dd28258-85867d745c-x8zfs\" (UID: \"1a008dc0-b654-4d0d-8440-fcc3e6f0def6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" Nov 02 23:00:28.206365 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:28.206154 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-maas-api-tls\") pod \"maas-api-e2e-ait-5dd28258-85867d745c-x8zfs\" (UID: \"1a008dc0-b654-4d0d-8440-fcc3e6f0def6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" Nov 02 23:00:28.206365 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:28.206176 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfj9r\" (UniqueName: \"kubernetes.io/projected/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-kube-api-access-nfj9r\") pod \"maas-api-e2e-ait-5dd28258-85867d745c-x8zfs\" (UID: \"1a008dc0-b654-4d0d-8440-fcc3e6f0def6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" Nov 02 23:00:28.206365 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:28.206309 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-openshift-service-ca\") pod \"maas-api-e2e-ait-5dd28258-85867d745c-x8zfs\" (UID: \"1a008dc0-b654-4d0d-8440-fcc3e6f0def6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" Nov 02 23:00:28.206587 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:00:28.206414 5114 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-serving-cert: secret "maas-api-e2e-ait-5dd28258-serving-cert" not found Nov 02 23:00:28.206587 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:00:28.206553 5114 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-maas-api-tls podName:1a008dc0-b654-4d0d-8440-fcc3e6f0def6 nodeName:}" failed. No retries permitted until 2026-11-02 23:00:28.706526977 +0000 UTC m=+704.059560355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-maas-api-tls") pod "maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" (UID: "1a008dc0-b654-4d0d-8440-fcc3e6f0def6") : secret "maas-api-e2e-ait-5dd28258-serving-cert" not found Nov 02 23:00:28.207117 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:28.207091 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-openshift-service-ca\") pod \"maas-api-e2e-ait-5dd28258-85867d745c-x8zfs\" (UID: \"1a008dc0-b654-4d0d-8440-fcc3e6f0def6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" Nov 02 23:00:28.208835 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:28.208812 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-metrics-certs\") pod \"maas-api-e2e-ait-5dd28258-85867d745c-x8zfs\" (UID: \"1a008dc0-b654-4d0d-8440-fcc3e6f0def6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" Nov 02 23:00:28.299822 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:28.299784 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfj9r\" (UniqueName: \"kubernetes.io/projected/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-kube-api-access-nfj9r\") pod \"maas-api-e2e-ait-5dd28258-85867d745c-x8zfs\" (UID: \"1a008dc0-b654-4d0d-8440-fcc3e6f0def6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" Nov 02 23:00:28.711633 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:28.711537 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-maas-api-tls\") pod \"maas-api-e2e-ait-5dd28258-85867d745c-x8zfs\" (UID: \"1a008dc0-b654-4d0d-8440-fcc3e6f0def6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" Nov 02 23:00:28.714284 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:28.714255 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-maas-api-tls\") pod \"maas-api-e2e-ait-5dd28258-85867d745c-x8zfs\" (UID: \"1a008dc0-b654-4d0d-8440-fcc3e6f0def6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" Nov 02 23:00:28.917067 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:28.917024 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" Nov 02 23:00:29.294536 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:29.294504 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs"] Nov 02 23:00:29.297890 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:00:29.297851 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a008dc0_b654_4d0d_8440_fcc3e6f0def6.slice/crio-90178bc888d6432e3e46f591624761e656318233b6e004d7160e24012b6caf2a WatchSource:0}: Error finding container 90178bc888d6432e3e46f591624761e656318233b6e004d7160e24012b6caf2a: Status 404 returned error can't find the container with id 90178bc888d6432e3e46f591624761e656318233b6e004d7160e24012b6caf2a Nov 02 23:00:29.588042 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:29.587955 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" event={"ID":"1a008dc0-b654-4d0d-8440-fcc3e6f0def6","Type":"ContainerStarted","Data":"90178bc888d6432e3e46f591624761e656318233b6e004d7160e24012b6caf2a"} Nov 02 23:00:31.598401 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:31.598357 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" event={"ID":"1a008dc0-b654-4d0d-8440-fcc3e6f0def6","Type":"ContainerStarted","Data":"51dd1b13517447cdccd7da4297287f383793e187db368eb7d1e4e53d98db0fd1"} Nov 02 23:00:31.598882 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:31.598524 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" Nov 02 23:00:31.717180 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:31.717126 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" podStartSLOduration=2.878006864 podStartE2EDuration="4.717110851s" totalImagesPullingTime="1.839103987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:00:31.716632343 +0000 UTC m=+707.069665728" watchObservedRunningTime="2026-11-02 23:00:31.717110851 +0000 UTC m=+707.070144262" Nov 02 23:00:33.575730 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:33.575693 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms"] Nov 02 23:00:33.579242 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:33.579221 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" Nov 02 23:00:33.631601 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:33.631563 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-fed9e8ee-serving-cert\"" Nov 02 23:00:33.657740 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:33.657703 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bded49bc-fc24-4fc2-ab5c-0b18129b1290-metrics-certs\") pod \"maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms\" (UID: \"bded49bc-fc24-4fc2-ab5c-0b18129b1290\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" Nov 02 23:00:33.657740 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:33.657741 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bded49bc-fc24-4fc2-ab5c-0b18129b1290-maas-api-tls\") pod \"maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms\" (UID: \"bded49bc-fc24-4fc2-ab5c-0b18129b1290\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" Nov 02 23:00:33.657993 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:33.657855 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55w4q\" (UniqueName: \"kubernetes.io/projected/bded49bc-fc24-4fc2-ab5c-0b18129b1290-kube-api-access-55w4q\") pod \"maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms\" (UID: \"bded49bc-fc24-4fc2-ab5c-0b18129b1290\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" Nov 02 23:00:33.657993 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:33.657964 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bded49bc-fc24-4fc2-ab5c-0b18129b1290-openshift-service-ca\") pod \"maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms\" (UID: \"bded49bc-fc24-4fc2-ab5c-0b18129b1290\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" Nov 02 23:00:33.683070 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:33.683036 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms"] Nov 02 23:00:33.759336 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:33.759299 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bded49bc-fc24-4fc2-ab5c-0b18129b1290-openshift-service-ca\") pod \"maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms\" (UID: \"bded49bc-fc24-4fc2-ab5c-0b18129b1290\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" Nov 02 23:00:33.759574 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:33.759339 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bded49bc-fc24-4fc2-ab5c-0b18129b1290-metrics-certs\") pod \"maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms\" (UID: \"bded49bc-fc24-4fc2-ab5c-0b18129b1290\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" Nov 02 23:00:33.759574 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:33.759370 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bded49bc-fc24-4fc2-ab5c-0b18129b1290-maas-api-tls\") pod \"maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms\" (UID: \"bded49bc-fc24-4fc2-ab5c-0b18129b1290\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" Nov 02 23:00:33.759574 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:33.759423 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55w4q\" (UniqueName: \"kubernetes.io/projected/bded49bc-fc24-4fc2-ab5c-0b18129b1290-kube-api-access-55w4q\") pod \"maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms\" (UID: \"bded49bc-fc24-4fc2-ab5c-0b18129b1290\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" Nov 02 23:00:33.759574 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:00:33.759540 5114 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-serving-cert: secret "maas-api-e2e-migrate-fed9e8ee-serving-cert" not found Nov 02 23:00:33.759806 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:00:33.759629 5114 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bded49bc-fc24-4fc2-ab5c-0b18129b1290-maas-api-tls podName:bded49bc-fc24-4fc2-ab5c-0b18129b1290 nodeName:}" failed. No retries permitted until 2026-11-02 23:00:34.259605767 +0000 UTC m=+709.612639124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bded49bc-fc24-4fc2-ab5c-0b18129b1290-maas-api-tls") pod "maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" (UID: "bded49bc-fc24-4fc2-ab5c-0b18129b1290") : secret "maas-api-e2e-migrate-fed9e8ee-serving-cert" not found Nov 02 23:00:33.760094 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:33.760076 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bded49bc-fc24-4fc2-ab5c-0b18129b1290-openshift-service-ca\") pod \"maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms\" (UID: \"bded49bc-fc24-4fc2-ab5c-0b18129b1290\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" Nov 02 23:00:33.762034 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:33.762016 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bded49bc-fc24-4fc2-ab5c-0b18129b1290-metrics-certs\") pod \"maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms\" (UID: \"bded49bc-fc24-4fc2-ab5c-0b18129b1290\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" Nov 02 23:00:33.823242 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:33.823213 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55w4q\" (UniqueName: \"kubernetes.io/projected/bded49bc-fc24-4fc2-ab5c-0b18129b1290-kube-api-access-55w4q\") pod \"maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms\" (UID: \"bded49bc-fc24-4fc2-ab5c-0b18129b1290\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" Nov 02 23:00:34.264975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:34.264934 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bded49bc-fc24-4fc2-ab5c-0b18129b1290-maas-api-tls\") pod \"maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms\" (UID: \"bded49bc-fc24-4fc2-ab5c-0b18129b1290\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" Nov 02 23:00:34.267753 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:34.267720 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bded49bc-fc24-4fc2-ab5c-0b18129b1290-maas-api-tls\") pod \"maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms\" (UID: \"bded49bc-fc24-4fc2-ab5c-0b18129b1290\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" Nov 02 23:00:34.492176 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:34.492128 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" Nov 02 23:00:34.866319 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:34.866283 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms"] Nov 02 23:00:34.869398 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:00:34.869356 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbded49bc_fc24_4fc2_ab5c_0b18129b1290.slice/crio-c9cde04147e367fb26ff26c2b3584cf4466d9efeeeecdede1fe97b04a6adedb0 WatchSource:0}: Error finding container c9cde04147e367fb26ff26c2b3584cf4466d9efeeeecdede1fe97b04a6adedb0: Status 404 returned error can't find the container with id c9cde04147e367fb26ff26c2b3584cf4466d9efeeeecdede1fe97b04a6adedb0 Nov 02 23:00:35.635232 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:35.635181 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" event={"ID":"bded49bc-fc24-4fc2-ab5c-0b18129b1290","Type":"ContainerStarted","Data":"c9cde04147e367fb26ff26c2b3584cf4466d9efeeeecdede1fe97b04a6adedb0"} Nov 02 23:00:36.641863 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:36.641825 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" event={"ID":"bded49bc-fc24-4fc2-ab5c-0b18129b1290","Type":"ContainerStarted","Data":"0bc626e1b8d50fdb46c773a942e8ff751fd8719b0aad52892bc3d6dd32099712"} Nov 02 23:00:36.642263 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:36.641909 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" Nov 02 23:00:36.758452 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:36.758384 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" podStartSLOduration=2.9551025920000003 podStartE2EDuration="3.758363286s" totalImagesPullingTime="803.260694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:00:36.757664316 +0000 UTC m=+712.110697704" watchObservedRunningTime="2026-11-02 23:00:36.758363286 +0000 UTC m=+712.111396662" Nov 02 23:00:37.584351 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:00:37.584313 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:00:37.634528 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:37.634498 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" Nov 02 23:00:42.651616 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:42.651585 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" Nov 02 23:00:44.683021 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:44.682990 5114 generic.go:356] "Generic (PLEG): container finished" podID="0dad888a-7ae1-481a-8f3d-877d6a1c52bf" containerID="fc48f2d69d86c43425ed0cc0c6d065bfaf15827b972204f4236b8d6c70c8c5b4" exitCode=0 Nov 02 23:00:44.683462 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:44.683042 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" event={"ID":"0dad888a-7ae1-481a-8f3d-877d6a1c52bf","Type":"ContainerDied","Data":"fc48f2d69d86c43425ed0cc0c6d065bfaf15827b972204f4236b8d6c70c8c5b4"} Nov 02 23:00:44.683462 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:44.683413 5114 scope.go:122] "RemoveContainer" containerID="fc48f2d69d86c43425ed0cc0c6d065bfaf15827b972204f4236b8d6c70c8c5b4" Nov 02 23:00:44.684763 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:44.684731 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:00:44.684841 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:44.684824 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:00:44.684893 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:44.684878 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:00:45.689080 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:45.689042 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" event={"ID":"0dad888a-7ae1-481a-8f3d-877d6a1c52bf","Type":"ContainerStarted","Data":"905e37288526849d13de6cdae1a2eeced48faa8d93a2969802e30d4dbe3d3213"} Nov 02 23:00:46.327509 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:46.326580 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:46.330508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:46.328544 5114 patch_prober.go:29] interesting pod/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 02 23:00:46.330508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:46.328604 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" podUID="0dad888a-7ae1-481a-8f3d-877d6a1c52bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 02 23:00:47.327509 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:47.322934 5114 patch_prober.go:29] interesting pod/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 02 23:00:47.327509 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:47.323001 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" podUID="0dad888a-7ae1-481a-8f3d-877d6a1c52bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 02 23:00:47.530681 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:47.530639 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:48.326000 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:48.325962 5114 patch_prober.go:29] interesting pod/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 02 23:00:48.326303 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:48.326267 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" podUID="0dad888a-7ae1-481a-8f3d-877d6a1c52bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 02 23:00:49.321761 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:49.321718 5114 patch_prober.go:29] interesting pod/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 02 23:00:49.330667 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:49.330598 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" podUID="0dad888a-7ae1-481a-8f3d-877d6a1c52bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 02 23:00:50.326511 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:50.324964 5114 patch_prober.go:29] interesting pod/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 02 23:00:50.326511 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:50.325048 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" podUID="0dad888a-7ae1-481a-8f3d-877d6a1c52bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 02 23:00:51.321510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:51.321459 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:00:51.323601 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:00:51.323571 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:01:07.584186 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:01:07.584139 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:01:08.132576 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:08.132516 5114 patch_prober.go:29] interesting pod/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": read tcp 10.134.0.2:35228->10.134.0.30:15021: read: connection reset by peer" start-of-body= Nov 02 23:01:08.132791 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:08.132640 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" podUID="d8eb89e7-8060-404a-b694-9d7bca211dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": read tcp 10.134.0.2:35228->10.134.0.30:15021: read: connection reset by peer" Nov 02 23:01:08.795538 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:08.795503 5114 generic.go:356] "Generic (PLEG): container finished" podID="d8eb89e7-8060-404a-b694-9d7bca211dac" containerID="de68994260b9c591da31a9bef0542c4fe61b04beb69013b7ab87337d1a74f906" exitCode=0 Nov 02 23:01:08.795947 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:08.795574 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" event={"ID":"d8eb89e7-8060-404a-b694-9d7bca211dac","Type":"ContainerDied","Data":"de68994260b9c591da31a9bef0542c4fe61b04beb69013b7ab87337d1a74f906"} Nov 02 23:01:08.795947 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:08.795933 5114 scope.go:122] "RemoveContainer" containerID="de68994260b9c591da31a9bef0542c4fe61b04beb69013b7ab87337d1a74f906" Nov 02 23:01:08.797321 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:08.797290 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:01:08.797443 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:08.797378 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:01:08.797443 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:08.797426 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:01:09.802011 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:09.801976 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" event={"ID":"d8eb89e7-8060-404a-b694-9d7bca211dac","Type":"ContainerStarted","Data":"087230b37145d08bbd7cd95f851107bb924ad0ff84665564348cca530642efd4"} Nov 02 23:01:10.119953 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:10.119409 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:01:10.119953 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:10.119576 5114 patch_prober.go:29] interesting pod/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Nov 02 23:01:10.119953 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:10.119623 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" podUID="d8eb89e7-8060-404a-b694-9d7bca211dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Nov 02 23:01:11.128999 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:11.128954 5114 patch_prober.go:29] interesting pod/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Nov 02 23:01:11.130085 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:11.129641 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" podUID="d8eb89e7-8060-404a-b694-9d7bca211dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Nov 02 23:01:12.116031 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:12.115983 5114 patch_prober.go:29] interesting pod/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Nov 02 23:01:12.116360 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:12.116323 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" podUID="d8eb89e7-8060-404a-b694-9d7bca211dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Nov 02 23:01:13.120516 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:13.116889 5114 patch_prober.go:29] interesting pod/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Nov 02 23:01:13.120516 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:13.116959 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" podUID="d8eb89e7-8060-404a-b694-9d7bca211dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Nov 02 23:01:14.125510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:14.117921 5114 patch_prober.go:29] interesting pod/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Nov 02 23:01:14.125510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:14.117988 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" podUID="d8eb89e7-8060-404a-b694-9d7bca211dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Nov 02 23:01:15.111678 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:15.111647 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:01:15.111962 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:15.111934 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:01:15.112853 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:15.112830 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:01:27.138846 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:27.138812 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs"] Nov 02 23:01:27.139416 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:27.139035 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" podUID="1a008dc0-b654-4d0d-8440-fcc3e6f0def6" containerName="maas-api" containerID="cri-o://51dd1b13517447cdccd7da4297287f383793e187db368eb7d1e4e53d98db0fd1" gracePeriod=30 Nov 02 23:01:27.411659 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:27.411632 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" Nov 02 23:01:27.507991 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:27.507949 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-metrics-certs\") pod \"1a008dc0-b654-4d0d-8440-fcc3e6f0def6\" (UID: \"1a008dc0-b654-4d0d-8440-fcc3e6f0def6\") " Nov 02 23:01:27.508200 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:27.508007 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-kube-api-access-nfj9r\" (UniqueName: \"kubernetes.io/projected/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-kube-api-access-nfj9r\") pod \"1a008dc0-b654-4d0d-8440-fcc3e6f0def6\" (UID: \"1a008dc0-b654-4d0d-8440-fcc3e6f0def6\") " Nov 02 23:01:27.508200 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:27.508031 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-maas-api-tls\") pod \"1a008dc0-b654-4d0d-8440-fcc3e6f0def6\" (UID: \"1a008dc0-b654-4d0d-8440-fcc3e6f0def6\") " Nov 02 23:01:27.508200 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:27.508055 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-openshift-service-ca\") pod \"1a008dc0-b654-4d0d-8440-fcc3e6f0def6\" (UID: \"1a008dc0-b654-4d0d-8440-fcc3e6f0def6\") " Nov 02 23:01:27.508568 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:27.508541 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-openshift-service-ca" pod "1a008dc0-b654-4d0d-8440-fcc3e6f0def6" (UID: "1a008dc0-b654-4d0d-8440-fcc3e6f0def6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:01:27.510294 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:27.510272 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-kube-api-access-nfj9r" pod "1a008dc0-b654-4d0d-8440-fcc3e6f0def6" (UID: "1a008dc0-b654-4d0d-8440-fcc3e6f0def6"). InnerVolumeSpecName "kube-api-access-nfj9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:27.510392 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:27.510317 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-maas-api-tls" pod "1a008dc0-b654-4d0d-8440-fcc3e6f0def6" (UID: "1a008dc0-b654-4d0d-8440-fcc3e6f0def6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:01:27.510392 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:27.510347 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-metrics-certs" pod "1a008dc0-b654-4d0d-8440-fcc3e6f0def6" (UID: "1a008dc0-b654-4d0d-8440-fcc3e6f0def6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:01:27.609246 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:27.609200 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-maas-api-tls\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:27.609246 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:27.609237 5114 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-openshift-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:27.609246 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:27.609247 5114 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-metrics-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:27.609246 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:27.609256 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfj9r\" (UniqueName: \"kubernetes.io/projected/1a008dc0-b654-4d0d-8440-fcc3e6f0def6-kube-api-access-nfj9r\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:27.903800 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:27.903759 5114 generic.go:356] "Generic (PLEG): container finished" podID="1a008dc0-b654-4d0d-8440-fcc3e6f0def6" containerID="51dd1b13517447cdccd7da4297287f383793e187db368eb7d1e4e53d98db0fd1" exitCode=0 Nov 02 23:01:27.903800 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:27.903799 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" Nov 02 23:01:27.904082 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:27.903845 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" event={"ID":"1a008dc0-b654-4d0d-8440-fcc3e6f0def6","Type":"ContainerDied","Data":"51dd1b13517447cdccd7da4297287f383793e187db368eb7d1e4e53d98db0fd1"} Nov 02 23:01:27.904082 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:27.903912 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs" event={"ID":"1a008dc0-b654-4d0d-8440-fcc3e6f0def6","Type":"ContainerDied","Data":"90178bc888d6432e3e46f591624761e656318233b6e004d7160e24012b6caf2a"} Nov 02 23:01:27.904082 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:27.903943 5114 scope.go:122] "RemoveContainer" containerID="51dd1b13517447cdccd7da4297287f383793e187db368eb7d1e4e53d98db0fd1" Nov 02 23:01:27.913777 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:27.913759 5114 scope.go:122] "RemoveContainer" containerID="51dd1b13517447cdccd7da4297287f383793e187db368eb7d1e4e53d98db0fd1" Nov 02 23:01:27.914091 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:01:27.914072 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51dd1b13517447cdccd7da4297287f383793e187db368eb7d1e4e53d98db0fd1\": container with ID starting with 51dd1b13517447cdccd7da4297287f383793e187db368eb7d1e4e53d98db0fd1 not found: ID does not exist" containerID="51dd1b13517447cdccd7da4297287f383793e187db368eb7d1e4e53d98db0fd1" Nov 02 23:01:27.914175 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:27.914105 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51dd1b13517447cdccd7da4297287f383793e187db368eb7d1e4e53d98db0fd1"} err="failed to get container status \"51dd1b13517447cdccd7da4297287f383793e187db368eb7d1e4e53d98db0fd1\": rpc error: code = NotFound desc = could not find container \"51dd1b13517447cdccd7da4297287f383793e187db368eb7d1e4e53d98db0fd1\": container with ID starting with 51dd1b13517447cdccd7da4297287f383793e187db368eb7d1e4e53d98db0fd1 not found: ID does not exist" Nov 02 23:01:28.066675 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:28.066642 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs"] Nov 02 23:01:28.070579 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:28.070551 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5dd28258-85867d745c-x8zfs"] Nov 02 23:01:28.435144 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:28.435107 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms"] Nov 02 23:01:28.435671 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:28.435307 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" podUID="bded49bc-fc24-4fc2-ab5c-0b18129b1290" containerName="maas-api" containerID="cri-o://0bc626e1b8d50fdb46c773a942e8ff751fd8719b0aad52892bc3d6dd32099712" gracePeriod=30 Nov 02 23:01:28.685343 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:28.685280 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" Nov 02 23:01:28.791778 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:28.791746 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a008dc0-b654-4d0d-8440-fcc3e6f0def6" path="/var/lib/kubelet/pods/1a008dc0-b654-4d0d-8440-fcc3e6f0def6/volumes" Nov 02 23:01:28.821315 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:28.821283 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bded49bc-fc24-4fc2-ab5c-0b18129b1290-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bded49bc-fc24-4fc2-ab5c-0b18129b1290-metrics-certs\") pod \"bded49bc-fc24-4fc2-ab5c-0b18129b1290\" (UID: \"bded49bc-fc24-4fc2-ab5c-0b18129b1290\") " Nov 02 23:01:28.821569 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:28.821323 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bded49bc-fc24-4fc2-ab5c-0b18129b1290-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bded49bc-fc24-4fc2-ab5c-0b18129b1290-openshift-service-ca\") pod \"bded49bc-fc24-4fc2-ab5c-0b18129b1290\" (UID: \"bded49bc-fc24-4fc2-ab5c-0b18129b1290\") " Nov 02 23:01:28.821569 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:28.821353 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bded49bc-fc24-4fc2-ab5c-0b18129b1290-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bded49bc-fc24-4fc2-ab5c-0b18129b1290-maas-api-tls\") pod \"bded49bc-fc24-4fc2-ab5c-0b18129b1290\" (UID: \"bded49bc-fc24-4fc2-ab5c-0b18129b1290\") " Nov 02 23:01:28.821569 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:28.821451 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bded49bc-fc24-4fc2-ab5c-0b18129b1290-kube-api-access-55w4q\" (UniqueName: \"kubernetes.io/projected/bded49bc-fc24-4fc2-ab5c-0b18129b1290-kube-api-access-55w4q\") pod \"bded49bc-fc24-4fc2-ab5c-0b18129b1290\" (UID: \"bded49bc-fc24-4fc2-ab5c-0b18129b1290\") " Nov 02 23:01:28.821815 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:28.821787 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bded49bc-fc24-4fc2-ab5c-0b18129b1290-openshift-service-ca" pod "bded49bc-fc24-4fc2-ab5c-0b18129b1290" (UID: "bded49bc-fc24-4fc2-ab5c-0b18129b1290"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:01:28.823852 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:28.823814 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bded49bc-fc24-4fc2-ab5c-0b18129b1290-metrics-certs" pod "bded49bc-fc24-4fc2-ab5c-0b18129b1290" (UID: "bded49bc-fc24-4fc2-ab5c-0b18129b1290"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:01:28.824007 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:28.823862 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bded49bc-fc24-4fc2-ab5c-0b18129b1290-maas-api-tls" pod "bded49bc-fc24-4fc2-ab5c-0b18129b1290" (UID: "bded49bc-fc24-4fc2-ab5c-0b18129b1290"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:01:28.824156 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:28.824140 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bded49bc-fc24-4fc2-ab5c-0b18129b1290-kube-api-access-55w4q" pod "bded49bc-fc24-4fc2-ab5c-0b18129b1290" (UID: "bded49bc-fc24-4fc2-ab5c-0b18129b1290"). InnerVolumeSpecName "kube-api-access-55w4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:28.910464 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:28.910424 5114 generic.go:356] "Generic (PLEG): container finished" podID="bded49bc-fc24-4fc2-ab5c-0b18129b1290" containerID="0bc626e1b8d50fdb46c773a942e8ff751fd8719b0aad52892bc3d6dd32099712" exitCode=0 Nov 02 23:01:28.910464 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:28.910468 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" Nov 02 23:01:28.910708 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:28.910513 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" event={"ID":"bded49bc-fc24-4fc2-ab5c-0b18129b1290","Type":"ContainerDied","Data":"0bc626e1b8d50fdb46c773a942e8ff751fd8719b0aad52892bc3d6dd32099712"} Nov 02 23:01:28.910708 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:28.910568 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms" event={"ID":"bded49bc-fc24-4fc2-ab5c-0b18129b1290","Type":"ContainerDied","Data":"c9cde04147e367fb26ff26c2b3584cf4466d9efeeeecdede1fe97b04a6adedb0"} Nov 02 23:01:28.910708 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:28.910592 5114 scope.go:122] "RemoveContainer" containerID="0bc626e1b8d50fdb46c773a942e8ff751fd8719b0aad52892bc3d6dd32099712" Nov 02 23:01:28.921453 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:28.921090 5114 scope.go:122] "RemoveContainer" containerID="0bc626e1b8d50fdb46c773a942e8ff751fd8719b0aad52892bc3d6dd32099712" Nov 02 23:01:28.922356 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:28.922269 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55w4q\" (UniqueName: \"kubernetes.io/projected/bded49bc-fc24-4fc2-ab5c-0b18129b1290-kube-api-access-55w4q\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:28.922356 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:28.922298 5114 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bded49bc-fc24-4fc2-ab5c-0b18129b1290-metrics-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:28.922356 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:28.922314 5114 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bded49bc-fc24-4fc2-ab5c-0b18129b1290-openshift-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:28.922356 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:28.922329 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bded49bc-fc24-4fc2-ab5c-0b18129b1290-maas-api-tls\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:28.922963 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:01:28.922724 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bc626e1b8d50fdb46c773a942e8ff751fd8719b0aad52892bc3d6dd32099712\": container with ID starting with 0bc626e1b8d50fdb46c773a942e8ff751fd8719b0aad52892bc3d6dd32099712 not found: ID does not exist" containerID="0bc626e1b8d50fdb46c773a942e8ff751fd8719b0aad52892bc3d6dd32099712" Nov 02 23:01:28.922963 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:28.922773 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc626e1b8d50fdb46c773a942e8ff751fd8719b0aad52892bc3d6dd32099712"} err="failed to get container status \"0bc626e1b8d50fdb46c773a942e8ff751fd8719b0aad52892bc3d6dd32099712\": rpc error: code = NotFound desc = could not find container \"0bc626e1b8d50fdb46c773a942e8ff751fd8719b0aad52892bc3d6dd32099712\": container with ID starting with 0bc626e1b8d50fdb46c773a942e8ff751fd8719b0aad52892bc3d6dd32099712 not found: ID does not exist" Nov 02 23:01:29.074597 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.074567 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms"] Nov 02 23:01:29.078658 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.078631 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fed9e8ee-69db7dbd67-mtdms"] Nov 02 23:01:29.505520 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.505417 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7"] Nov 02 23:01:29.505999 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.505953 5114 cpu_manager.go:414] "containerMap: removing container" podUID="1a008dc0-b654-4d0d-8440-fcc3e6f0def6" containerName="maas-api" Nov 02 23:01:29.505999 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.505974 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a008dc0-b654-4d0d-8440-fcc3e6f0def6" containerName="maas-api" Nov 02 23:01:29.505999 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.505996 5114 cpu_manager.go:414] "containerMap: removing container" podUID="bded49bc-fc24-4fc2-ab5c-0b18129b1290" containerName="maas-api" Nov 02 23:01:29.506155 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.506003 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bded49bc-fc24-4fc2-ab5c-0b18129b1290" containerName="maas-api" Nov 02 23:01:29.506155 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.506094 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a008dc0-b654-4d0d-8440-fcc3e6f0def6" containerName="maas-api" Nov 02 23:01:29.506155 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.506108 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="bded49bc-fc24-4fc2-ab5c-0b18129b1290" containerName="maas-api" Nov 02 23:01:29.508542 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.508521 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.563529 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.563495 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-08c2c6-openshift-default-dockercfg-qcc5b\"" Nov 02 23:01:29.595517 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.590048 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7"] Nov 02 23:01:29.629079 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.629036 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzn4t\" (UniqueName: \"kubernetes.io/projected/d2cc2d36-def2-45fb-86bf-b35f56a969e6-kube-api-access-qzn4t\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.629079 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.629081 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.629347 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.629238 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.629347 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.629294 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.629435 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.629371 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.629435 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.629428 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.629572 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.629445 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.629572 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.629464 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.629676 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.629629 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.730263 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.730229 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzn4t\" (UniqueName: \"kubernetes.io/projected/d2cc2d36-def2-45fb-86bf-b35f56a969e6-kube-api-access-qzn4t\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.730263 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.730271 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.730571 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.730329 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.730571 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.730347 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.730571 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.730366 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.730571 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.730414 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.730571 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.730433 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.730571 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.730454 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.730571 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.730506 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.730931 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.730780 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.730931 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.730906 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.731041 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.731013 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.731103 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.731086 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.731593 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.731558 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.733177 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.733155 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.733590 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.733569 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.791112 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.791081 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.791264 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.791201 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzn4t\" (UniqueName: \"kubernetes.io/projected/d2cc2d36-def2-45fb-86bf-b35f56a969e6-kube-api-access-qzn4t\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.819903 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.819861 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:29.985005 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.984966 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm"] Nov 02 23:01:29.985293 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:29.985252 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" podUID="0dad888a-7ae1-481a-8f3d-877d6a1c52bf" containerName="istio-proxy" containerID="cri-o://905e37288526849d13de6cdae1a2eeced48faa8d93a2969802e30d4dbe3d3213" gracePeriod=30 Nov 02 23:01:30.243824 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:30.243712 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7"] Nov 02 23:01:30.246420 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:01:30.246388 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2cc2d36_def2_45fb_86bf_b35f56a969e6.slice/crio-bcdfd16844dc8de03beca527fe0dd26b00ec915a0ee11d0aaf7c6b175cab1225 WatchSource:0}: Error finding container bcdfd16844dc8de03beca527fe0dd26b00ec915a0ee11d0aaf7c6b175cab1225: Status 404 returned error can't find the container with id bcdfd16844dc8de03beca527fe0dd26b00ec915a0ee11d0aaf7c6b175cab1225 Nov 02 23:01:30.248651 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:30.248610 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:01:30.248733 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:30.248681 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:01:30.248733 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:30.248715 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:01:30.795060 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:30.793007 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bded49bc-fc24-4fc2-ab5c-0b18129b1290" path="/var/lib/kubelet/pods/bded49bc-fc24-4fc2-ab5c-0b18129b1290/volumes" Nov 02 23:01:30.921850 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:30.921807 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" event={"ID":"d2cc2d36-def2-45fb-86bf-b35f56a969e6","Type":"ContainerStarted","Data":"a761c07dd672a89d09871c4ad763be5755bed6ad2a655a8a8b300f3a6323eb8b"} Nov 02 23:01:30.922039 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:30.921871 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" event={"ID":"d2cc2d36-def2-45fb-86bf-b35f56a969e6","Type":"ContainerStarted","Data":"bcdfd16844dc8de03beca527fe0dd26b00ec915a0ee11d0aaf7c6b175cab1225"} Nov 02 23:01:31.071437 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:31.071323 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" podStartSLOduration=2.071306837 podStartE2EDuration="2.071306837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:31.070222286 +0000 UTC m=+766.423255663" watchObservedRunningTime="2026-11-02 23:01:31.071306837 +0000 UTC m=+766.424340213" Nov 02 23:01:31.820291 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:31.820249 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:31.825688 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:31.825656 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:31.927149 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:31.927106 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:31.928146 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:31.928124 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:01:32.243141 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.243045 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9"] Nov 02 23:01:32.245625 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.245603 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.299063 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.299033 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-96031a-openshift-default-dockercfg-fx8gh\"" Nov 02 23:01:32.311694 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.311656 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9"] Nov 02 23:01:32.353305 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.353251 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-workload-certs\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.353305 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.353304 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-token\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.353664 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.353409 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jxbb\" (UniqueName: \"kubernetes.io/projected/57299a9e-ed21-49ee-acdd-81831c2efd84-kube-api-access-7jxbb\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.353664 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.353447 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57299a9e-ed21-49ee-acdd-81831c2efd84-istiod-ca-cert\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.353664 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.353542 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-envoy\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.353664 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.353581 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-data\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.353882 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.353720 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-credential-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.353882 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.353770 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-workload-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.353882 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.353789 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-podinfo\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.454250 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.454216 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-workload-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.454250 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.454252 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-podinfo\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.454541 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.454306 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-workload-certs\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.454541 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.454335 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-token\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.454541 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.454369 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jxbb\" (UniqueName: \"kubernetes.io/projected/57299a9e-ed21-49ee-acdd-81831c2efd84-kube-api-access-7jxbb\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.454541 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.454399 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57299a9e-ed21-49ee-acdd-81831c2efd84-istiod-ca-cert\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.454541 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.454449 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-envoy\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.454541 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.454521 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-data\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.454834 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.454603 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-credential-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.454834 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.454668 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-workload-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.454834 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.454691 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-workload-certs\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.454953 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.454896 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-credential-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.455036 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.455012 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-data\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.455160 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.455132 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57299a9e-ed21-49ee-acdd-81831c2efd84-istiod-ca-cert\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.457044 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.457008 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-envoy\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.457536 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.457511 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-podinfo\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.515711 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.515621 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-token\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.515884 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.515726 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jxbb\" (UniqueName: \"kubernetes.io/projected/57299a9e-ed21-49ee-acdd-81831c2efd84-kube-api-access-7jxbb\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.558997 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.558951 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:32.845344 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.845318 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9"] Nov 02 23:01:32.847885 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:01:32.847849 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57299a9e_ed21_49ee_acdd_81831c2efd84.slice/crio-9b8456b27b571219b6e1e6cd8ea2b4550c69ee2f3d7db4e2bfb74c10930f6876 WatchSource:0}: Error finding container 9b8456b27b571219b6e1e6cd8ea2b4550c69ee2f3d7db4e2bfb74c10930f6876: Status 404 returned error can't find the container with id 9b8456b27b571219b6e1e6cd8ea2b4550c69ee2f3d7db4e2bfb74c10930f6876 Nov 02 23:01:32.850328 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.850295 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:01:32.850415 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.850365 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:01:32.850415 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.850393 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:01:32.933857 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.933800 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" event={"ID":"57299a9e-ed21-49ee-acdd-81831c2efd84","Type":"ContainerStarted","Data":"4aaca3c96642a52203751a35e36c120e36fcb7f70ecee0cbd6cc4e9433c4a40b"} Nov 02 23:01:32.934045 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:32.933893 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" event={"ID":"57299a9e-ed21-49ee-acdd-81831c2efd84","Type":"ContainerStarted","Data":"9b8456b27b571219b6e1e6cd8ea2b4550c69ee2f3d7db4e2bfb74c10930f6876"} Nov 02 23:01:33.054753 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:33.054697 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" podStartSLOduration=1.054678659 podStartE2EDuration="1.054678659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:33.052938974 +0000 UTC m=+768.405972348" watchObservedRunningTime="2026-11-02 23:01:33.054678659 +0000 UTC m=+768.407712034" Nov 02 23:01:33.559378 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:33.559327 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:33.565132 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:33.565100 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:33.938391 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:33.938303 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:33.939210 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:33.939190 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:01:35.248320 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.248245 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:01:35.384699 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.384660 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-credential-socket\") pod \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " Nov 02 23:01:35.384889 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.384726 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-envoy\") pod \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " Nov 02 23:01:35.384889 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.384781 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-kube-api-access-x6xpf\" (UniqueName: \"kubernetes.io/projected/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-kube-api-access-x6xpf\") pod \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " Nov 02 23:01:35.384889 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.384808 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-workload-certs\") pod \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " Nov 02 23:01:35.384889 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.384882 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-podinfo\") pod \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " Nov 02 23:01:35.385768 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.384911 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-data\") pod \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " Nov 02 23:01:35.385768 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.384939 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-token\" (UniqueName: \"kubernetes.io/projected/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-token\") pod \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " Nov 02 23:01:35.385768 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.384975 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istiod-ca-cert\") pod \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " Nov 02 23:01:35.385768 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.384984 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-credential-socket" pod "0dad888a-7ae1-481a-8f3d-877d6a1c52bf" (UID: "0dad888a-7ae1-481a-8f3d-877d6a1c52bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:01:35.385768 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.385072 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-workload-socket\") pod \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\" (UID: \"0dad888a-7ae1-481a-8f3d-877d6a1c52bf\") " Nov 02 23:01:35.385768 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.385098 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-workload-certs" pod "0dad888a-7ae1-481a-8f3d-877d6a1c52bf" (UID: "0dad888a-7ae1-481a-8f3d-877d6a1c52bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:01:35.385768 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.385159 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-data" pod "0dad888a-7ae1-481a-8f3d-877d6a1c52bf" (UID: "0dad888a-7ae1-481a-8f3d-877d6a1c52bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:01:35.385768 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.385334 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-workload-socket" pod "0dad888a-7ae1-481a-8f3d-877d6a1c52bf" (UID: "0dad888a-7ae1-481a-8f3d-877d6a1c52bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:01:35.385768 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.385417 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istiod-ca-cert" pod "0dad888a-7ae1-481a-8f3d-877d6a1c52bf" (UID: "0dad888a-7ae1-481a-8f3d-877d6a1c52bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:01:35.385768 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.385450 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:35.385768 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.385472 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:35.385768 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.385581 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:35.385768 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.385597 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:35.387299 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.387269 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-envoy" pod "0dad888a-7ae1-481a-8f3d-877d6a1c52bf" (UID: "0dad888a-7ae1-481a-8f3d-877d6a1c52bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:01:35.387428 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.387314 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-token" pod "0dad888a-7ae1-481a-8f3d-877d6a1c52bf" (UID: "0dad888a-7ae1-481a-8f3d-877d6a1c52bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:35.387705 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.387688 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-kube-api-access-x6xpf" pod "0dad888a-7ae1-481a-8f3d-877d6a1c52bf" (UID: "0dad888a-7ae1-481a-8f3d-877d6a1c52bf"). InnerVolumeSpecName "kube-api-access-x6xpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:35.387777 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.387761 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-podinfo" pod "0dad888a-7ae1-481a-8f3d-877d6a1c52bf" (UID: "0dad888a-7ae1-481a-8f3d-877d6a1c52bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:01:35.486378 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.486343 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:35.486378 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.486376 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6xpf\" (UniqueName: \"kubernetes.io/projected/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-kube-api-access-x6xpf\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:35.486378 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.486386 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:35.486721 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.486395 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:35.486721 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.486403 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dad888a-7ae1-481a-8f3d-877d6a1c52bf-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:35.949803 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.949766 5114 generic.go:356] "Generic (PLEG): container finished" podID="0dad888a-7ae1-481a-8f3d-877d6a1c52bf" containerID="905e37288526849d13de6cdae1a2eeced48faa8d93a2969802e30d4dbe3d3213" exitCode=0 Nov 02 23:01:35.950034 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.949819 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" Nov 02 23:01:35.950034 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.949859 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" event={"ID":"0dad888a-7ae1-481a-8f3d-877d6a1c52bf","Type":"ContainerDied","Data":"905e37288526849d13de6cdae1a2eeced48faa8d93a2969802e30d4dbe3d3213"} Nov 02 23:01:35.950034 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.949929 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm" event={"ID":"0dad888a-7ae1-481a-8f3d-877d6a1c52bf","Type":"ContainerDied","Data":"b8fba98b7a862265304b987c1719ab2255eace79ab8347063a372ede93633c20"} Nov 02 23:01:35.950034 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.949959 5114 scope.go:122] "RemoveContainer" containerID="905e37288526849d13de6cdae1a2eeced48faa8d93a2969802e30d4dbe3d3213" Nov 02 23:01:35.959868 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.959843 5114 scope.go:122] "RemoveContainer" containerID="fc48f2d69d86c43425ed0cc0c6d065bfaf15827b972204f4236b8d6c70c8c5b4" Nov 02 23:01:35.968886 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.968866 5114 scope.go:122] "RemoveContainer" containerID="905e37288526849d13de6cdae1a2eeced48faa8d93a2969802e30d4dbe3d3213" Nov 02 23:01:35.969221 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:01:35.969204 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"905e37288526849d13de6cdae1a2eeced48faa8d93a2969802e30d4dbe3d3213\": container with ID starting with 905e37288526849d13de6cdae1a2eeced48faa8d93a2969802e30d4dbe3d3213 not found: ID does not exist" containerID="905e37288526849d13de6cdae1a2eeced48faa8d93a2969802e30d4dbe3d3213" Nov 02 23:01:35.969276 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.969231 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"905e37288526849d13de6cdae1a2eeced48faa8d93a2969802e30d4dbe3d3213"} err="failed to get container status \"905e37288526849d13de6cdae1a2eeced48faa8d93a2969802e30d4dbe3d3213\": rpc error: code = NotFound desc = could not find container \"905e37288526849d13de6cdae1a2eeced48faa8d93a2969802e30d4dbe3d3213\": container with ID starting with 905e37288526849d13de6cdae1a2eeced48faa8d93a2969802e30d4dbe3d3213 not found: ID does not exist" Nov 02 23:01:35.969276 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.969247 5114 scope.go:122] "RemoveContainer" containerID="fc48f2d69d86c43425ed0cc0c6d065bfaf15827b972204f4236b8d6c70c8c5b4" Nov 02 23:01:35.969535 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:01:35.969516 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc48f2d69d86c43425ed0cc0c6d065bfaf15827b972204f4236b8d6c70c8c5b4\": container with ID starting with fc48f2d69d86c43425ed0cc0c6d065bfaf15827b972204f4236b8d6c70c8c5b4 not found: ID does not exist" containerID="fc48f2d69d86c43425ed0cc0c6d065bfaf15827b972204f4236b8d6c70c8c5b4" Nov 02 23:01:35.969600 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:35.969548 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc48f2d69d86c43425ed0cc0c6d065bfaf15827b972204f4236b8d6c70c8c5b4"} err="failed to get container status \"fc48f2d69d86c43425ed0cc0c6d065bfaf15827b972204f4236b8d6c70c8c5b4\": rpc error: code = NotFound desc = could not find container \"fc48f2d69d86c43425ed0cc0c6d065bfaf15827b972204f4236b8d6c70c8c5b4\": container with ID starting with fc48f2d69d86c43425ed0cc0c6d065bfaf15827b972204f4236b8d6c70c8c5b4 not found: ID does not exist" Nov 02 23:01:36.114071 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:36.114034 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm"] Nov 02 23:01:36.122298 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:36.122271 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5dd28258-openshift-default-cf488d589-d8lbm"] Nov 02 23:01:36.792401 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:36.792361 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dad888a-7ae1-481a-8f3d-877d6a1c52bf" path="/var/lib/kubelet/pods/0dad888a-7ae1-481a-8f3d-877d6a1c52bf/volumes" Nov 02 23:01:37.583734 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:01:37.583693 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:01:58.396722 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:58.396671 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv"] Nov 02 23:01:58.397222 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:58.397041 5114 cpu_manager.go:414] "containerMap: removing container" podUID="0dad888a-7ae1-481a-8f3d-877d6a1c52bf" containerName="istio-proxy" Nov 02 23:01:58.397222 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:58.397054 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dad888a-7ae1-481a-8f3d-877d6a1c52bf" containerName="istio-proxy" Nov 02 23:01:58.397222 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:58.397130 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dad888a-7ae1-481a-8f3d-877d6a1c52bf" containerName="istio-proxy" Nov 02 23:01:58.397222 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:58.397138 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dad888a-7ae1-481a-8f3d-877d6a1c52bf" containerName="istio-proxy" Nov 02 23:01:58.400184 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:58.400161 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" Nov 02 23:01:58.452082 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:58.452039 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-96031a-serving-cert\"" Nov 02 23:01:58.477663 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:58.477624 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/171d9e93-d16f-4da2-beb7-4526082dbc3f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv\" (UID: \"171d9e93-d16f-4da2-beb7-4526082dbc3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" Nov 02 23:01:58.477848 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:58.477676 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/171d9e93-d16f-4da2-beb7-4526082dbc3f-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv\" (UID: \"171d9e93-d16f-4da2-beb7-4526082dbc3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" Nov 02 23:01:58.477848 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:58.477734 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/171d9e93-d16f-4da2-beb7-4526082dbc3f-metrics-certs\") pod \"maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv\" (UID: \"171d9e93-d16f-4da2-beb7-4526082dbc3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" Nov 02 23:01:58.477848 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:58.477807 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47nmp\" (UniqueName: \"kubernetes.io/projected/171d9e93-d16f-4da2-beb7-4526082dbc3f-kube-api-access-47nmp\") pod \"maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv\" (UID: \"171d9e93-d16f-4da2-beb7-4526082dbc3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" Nov 02 23:01:58.502846 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:58.502813 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv"] Nov 02 23:01:58.578649 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:58.578606 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/171d9e93-d16f-4da2-beb7-4526082dbc3f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv\" (UID: \"171d9e93-d16f-4da2-beb7-4526082dbc3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" Nov 02 23:01:58.578855 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:58.578672 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/171d9e93-d16f-4da2-beb7-4526082dbc3f-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv\" (UID: \"171d9e93-d16f-4da2-beb7-4526082dbc3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" Nov 02 23:01:58.578855 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:58.578710 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/171d9e93-d16f-4da2-beb7-4526082dbc3f-metrics-certs\") pod \"maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv\" (UID: \"171d9e93-d16f-4da2-beb7-4526082dbc3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" Nov 02 23:01:58.578855 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:01:58.578765 5114 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-serving-cert: secret "maas-api-e2e-worker-w3-96031a-serving-cert" not found Nov 02 23:01:58.578855 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:58.578785 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47nmp\" (UniqueName: \"kubernetes.io/projected/171d9e93-d16f-4da2-beb7-4526082dbc3f-kube-api-access-47nmp\") pod \"maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv\" (UID: \"171d9e93-d16f-4da2-beb7-4526082dbc3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" Nov 02 23:01:58.579086 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:01:58.578905 5114 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/171d9e93-d16f-4da2-beb7-4526082dbc3f-maas-api-tls podName:171d9e93-d16f-4da2-beb7-4526082dbc3f nodeName:}" failed. No retries permitted until 2026-11-02 23:01:59.078827706 +0000 UTC m=+794.431861072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/171d9e93-d16f-4da2-beb7-4526082dbc3f-maas-api-tls") pod "maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" (UID: "171d9e93-d16f-4da2-beb7-4526082dbc3f") : secret "maas-api-e2e-worker-w3-96031a-serving-cert" not found Nov 02 23:01:58.579419 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:58.579393 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/171d9e93-d16f-4da2-beb7-4526082dbc3f-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv\" (UID: \"171d9e93-d16f-4da2-beb7-4526082dbc3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" Nov 02 23:01:58.581426 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:58.581407 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/171d9e93-d16f-4da2-beb7-4526082dbc3f-metrics-certs\") pod \"maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv\" (UID: \"171d9e93-d16f-4da2-beb7-4526082dbc3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" Nov 02 23:01:58.640313 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:58.640253 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47nmp\" (UniqueName: \"kubernetes.io/projected/171d9e93-d16f-4da2-beb7-4526082dbc3f-kube-api-access-47nmp\") pod \"maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv\" (UID: \"171d9e93-d16f-4da2-beb7-4526082dbc3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" Nov 02 23:01:59.083119 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:59.083087 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/171d9e93-d16f-4da2-beb7-4526082dbc3f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv\" (UID: \"171d9e93-d16f-4da2-beb7-4526082dbc3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" Nov 02 23:01:59.085787 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:59.085768 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/171d9e93-d16f-4da2-beb7-4526082dbc3f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv\" (UID: \"171d9e93-d16f-4da2-beb7-4526082dbc3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" Nov 02 23:01:59.330769 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:59.330713 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" Nov 02 23:01:59.730658 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:01:59.730529 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv"] Nov 02 23:01:59.738633 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:01:59.738560 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod171d9e93_d16f_4da2_beb7_4526082dbc3f.slice/crio-01bf769b54084c3a91ba256805969c82bd8835daeb35daa31d7e29df0769f6ba WatchSource:0}: Error finding container 01bf769b54084c3a91ba256805969c82bd8835daeb35daa31d7e29df0769f6ba: Status 404 returned error can't find the container with id 01bf769b54084c3a91ba256805969c82bd8835daeb35daa31d7e29df0769f6ba Nov 02 23:02:00.079569 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:00.079533 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" event={"ID":"171d9e93-d16f-4da2-beb7-4526082dbc3f","Type":"ContainerStarted","Data":"01bf769b54084c3a91ba256805969c82bd8835daeb35daa31d7e29df0769f6ba"} Nov 02 23:02:01.085882 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:01.085841 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" event={"ID":"171d9e93-d16f-4da2-beb7-4526082dbc3f","Type":"ContainerStarted","Data":"e1b6bde4d88e0d4da33e0a9b2f0bdf9e4a67e1915f93e84f94ff631a10e4c6a4"} Nov 02 23:02:01.086325 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:01.085949 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" Nov 02 23:02:01.204435 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:01.204379 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" podStartSLOduration=2.363239073 podStartE2EDuration="3.204362728s" totalImagesPullingTime="841.123655ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:01.203897605 +0000 UTC m=+796.556930981" watchObservedRunningTime="2026-11-02 23:02:01.204362728 +0000 UTC m=+796.557396147" Nov 02 23:02:05.207058 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:05.207010 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6"] Nov 02 23:02:05.207613 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:05.207295 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" podUID="d8eb89e7-8060-404a-b694-9d7bca211dac" containerName="istio-proxy" containerID="cri-o://087230b37145d08bbd7cd95f851107bb924ad0ff84665564348cca530642efd4" gracePeriod=30 Nov 02 23:02:07.095749 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:07.095718 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" Nov 02 23:02:07.583590 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:02:07.583464 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:02:08.134909 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:08.134874 5114 generic.go:356] "Generic (PLEG): container finished" podID="d2cc2d36-def2-45fb-86bf-b35f56a969e6" containerID="a761c07dd672a89d09871c4ad763be5755bed6ad2a655a8a8b300f3a6323eb8b" exitCode=0 Nov 02 23:02:08.135337 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:08.134950 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" event={"ID":"d2cc2d36-def2-45fb-86bf-b35f56a969e6","Type":"ContainerDied","Data":"a761c07dd672a89d09871c4ad763be5755bed6ad2a655a8a8b300f3a6323eb8b"} Nov 02 23:02:08.135393 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:08.135367 5114 scope.go:122] "RemoveContainer" containerID="a761c07dd672a89d09871c4ad763be5755bed6ad2a655a8a8b300f3a6323eb8b" Nov 02 23:02:08.136697 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:08.136664 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:02:08.136829 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:08.136755 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:02:08.136829 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:08.136808 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:02:09.152812 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:09.152776 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" event={"ID":"d2cc2d36-def2-45fb-86bf-b35f56a969e6","Type":"ContainerStarted","Data":"589c21f68ad0ee71280b448cf87e25e326a3bda5d18b78f7cd3b5e658662fc6b"} Nov 02 23:02:09.823976 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:09.823938 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:02:09.824413 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:09.824301 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 02 23:02:09.824616 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:09.824590 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" podUID="d2cc2d36-def2-45fb-86bf-b35f56a969e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 02 23:02:10.582342 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.582067 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:02:10.655777 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.655729 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-token\" (UniqueName: \"kubernetes.io/projected/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-token\") pod \"d8eb89e7-8060-404a-b694-9d7bca211dac\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " Nov 02 23:02:10.655986 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.655797 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8eb89e7-8060-404a-b694-9d7bca211dac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8eb89e7-8060-404a-b694-9d7bca211dac-istiod-ca-cert\") pod \"d8eb89e7-8060-404a-b694-9d7bca211dac\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " Nov 02 23:02:10.655986 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.655830 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8eb89e7-8060-404a-b694-9d7bca211dac-kube-api-access-2cxpz\" (UniqueName: \"kubernetes.io/projected/d8eb89e7-8060-404a-b694-9d7bca211dac-kube-api-access-2cxpz\") pod \"d8eb89e7-8060-404a-b694-9d7bca211dac\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " Nov 02 23:02:10.655986 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.655885 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-credential-socket\") pod \"d8eb89e7-8060-404a-b694-9d7bca211dac\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " Nov 02 23:02:10.655986 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.655976 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-data\") pod \"d8eb89e7-8060-404a-b694-9d7bca211dac\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " Nov 02 23:02:10.656212 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.656008 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-workload-certs\") pod \"d8eb89e7-8060-404a-b694-9d7bca211dac\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " Nov 02 23:02:10.656212 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.656046 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-workload-socket\") pod \"d8eb89e7-8060-404a-b694-9d7bca211dac\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " Nov 02 23:02:10.656212 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.656098 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-envoy\") pod \"d8eb89e7-8060-404a-b694-9d7bca211dac\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " Nov 02 23:02:10.656212 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.656132 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-podinfo\") pod \"d8eb89e7-8060-404a-b694-9d7bca211dac\" (UID: \"d8eb89e7-8060-404a-b694-9d7bca211dac\") " Nov 02 23:02:10.656882 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.656828 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-workload-certs" pod "d8eb89e7-8060-404a-b694-9d7bca211dac" (UID: "d8eb89e7-8060-404a-b694-9d7bca211dac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:02:10.657057 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.656852 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-credential-socket" pod "d8eb89e7-8060-404a-b694-9d7bca211dac" (UID: "d8eb89e7-8060-404a-b694-9d7bca211dac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:02:10.657169 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.656902 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8eb89e7-8060-404a-b694-9d7bca211dac-istiod-ca-cert" pod "d8eb89e7-8060-404a-b694-9d7bca211dac" (UID: "d8eb89e7-8060-404a-b694-9d7bca211dac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:02:10.657253 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.656890 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-workload-socket" pod "d8eb89e7-8060-404a-b694-9d7bca211dac" (UID: "d8eb89e7-8060-404a-b694-9d7bca211dac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:02:10.657350 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.657255 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-data" pod "d8eb89e7-8060-404a-b694-9d7bca211dac" (UID: "d8eb89e7-8060-404a-b694-9d7bca211dac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:02:10.659531 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.659504 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-envoy" pod "d8eb89e7-8060-404a-b694-9d7bca211dac" (UID: "d8eb89e7-8060-404a-b694-9d7bca211dac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:02:10.659769 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.659742 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-token" pod "d8eb89e7-8060-404a-b694-9d7bca211dac" (UID: "d8eb89e7-8060-404a-b694-9d7bca211dac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:02:10.659846 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.659812 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-podinfo" pod "d8eb89e7-8060-404a-b694-9d7bca211dac" (UID: "d8eb89e7-8060-404a-b694-9d7bca211dac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:02:10.660631 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.660588 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8eb89e7-8060-404a-b694-9d7bca211dac-kube-api-access-2cxpz" pod "d8eb89e7-8060-404a-b694-9d7bca211dac" (UID: "d8eb89e7-8060-404a-b694-9d7bca211dac"). InnerVolumeSpecName "kube-api-access-2cxpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:02:10.756896 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.756860 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:02:10.756896 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.756892 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:02:10.756896 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.756902 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:02:10.757159 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.756912 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:02:10.757159 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.756920 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:02:10.757159 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.756929 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:02:10.757159 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.756937 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8eb89e7-8060-404a-b694-9d7bca211dac-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:02:10.757159 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.756945 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8eb89e7-8060-404a-b694-9d7bca211dac-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:02:10.757159 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.756954 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cxpz\" (UniqueName: \"kubernetes.io/projected/d8eb89e7-8060-404a-b694-9d7bca211dac-kube-api-access-2cxpz\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:02:10.825570 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.822911 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 02 23:02:10.825570 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:10.822977 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" podUID="d2cc2d36-def2-45fb-86bf-b35f56a969e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 02 23:02:11.166192 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:11.166104 5114 generic.go:356] "Generic (PLEG): container finished" podID="d8eb89e7-8060-404a-b694-9d7bca211dac" containerID="087230b37145d08bbd7cd95f851107bb924ad0ff84665564348cca530642efd4" exitCode=0 Nov 02 23:02:11.166192 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:11.166144 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" event={"ID":"d8eb89e7-8060-404a-b694-9d7bca211dac","Type":"ContainerDied","Data":"087230b37145d08bbd7cd95f851107bb924ad0ff84665564348cca530642efd4"} Nov 02 23:02:11.166192 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:11.166185 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" event={"ID":"d8eb89e7-8060-404a-b694-9d7bca211dac","Type":"ContainerDied","Data":"2a61f275a58f79f841ad420d40e34ddd1c8381d7fa5584a3cba6b3f0aca89fed"} Nov 02 23:02:11.166413 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:11.166206 5114 scope.go:122] "RemoveContainer" containerID="087230b37145d08bbd7cd95f851107bb924ad0ff84665564348cca530642efd4" Nov 02 23:02:11.166413 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:11.166331 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6" Nov 02 23:02:11.176178 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:11.176160 5114 scope.go:122] "RemoveContainer" containerID="de68994260b9c591da31a9bef0542c4fe61b04beb69013b7ab87337d1a74f906" Nov 02 23:02:11.186797 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:11.186776 5114 scope.go:122] "RemoveContainer" containerID="087230b37145d08bbd7cd95f851107bb924ad0ff84665564348cca530642efd4" Nov 02 23:02:11.187143 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:02:11.187123 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"087230b37145d08bbd7cd95f851107bb924ad0ff84665564348cca530642efd4\": container with ID starting with 087230b37145d08bbd7cd95f851107bb924ad0ff84665564348cca530642efd4 not found: ID does not exist" containerID="087230b37145d08bbd7cd95f851107bb924ad0ff84665564348cca530642efd4" Nov 02 23:02:11.187197 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:11.187154 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087230b37145d08bbd7cd95f851107bb924ad0ff84665564348cca530642efd4"} err="failed to get container status \"087230b37145d08bbd7cd95f851107bb924ad0ff84665564348cca530642efd4\": rpc error: code = NotFound desc = could not find container \"087230b37145d08bbd7cd95f851107bb924ad0ff84665564348cca530642efd4\": container with ID starting with 087230b37145d08bbd7cd95f851107bb924ad0ff84665564348cca530642efd4 not found: ID does not exist" Nov 02 23:02:11.187197 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:11.187175 5114 scope.go:122] "RemoveContainer" containerID="de68994260b9c591da31a9bef0542c4fe61b04beb69013b7ab87337d1a74f906" Nov 02 23:02:11.187437 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:02:11.187421 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de68994260b9c591da31a9bef0542c4fe61b04beb69013b7ab87337d1a74f906\": container with ID starting with de68994260b9c591da31a9bef0542c4fe61b04beb69013b7ab87337d1a74f906 not found: ID does not exist" containerID="de68994260b9c591da31a9bef0542c4fe61b04beb69013b7ab87337d1a74f906" Nov 02 23:02:11.187523 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:11.187440 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de68994260b9c591da31a9bef0542c4fe61b04beb69013b7ab87337d1a74f906"} err="failed to get container status \"de68994260b9c591da31a9bef0542c4fe61b04beb69013b7ab87337d1a74f906\": rpc error: code = NotFound desc = could not find container \"de68994260b9c591da31a9bef0542c4fe61b04beb69013b7ab87337d1a74f906\": container with ID starting with de68994260b9c591da31a9bef0542c4fe61b04beb69013b7ab87337d1a74f906 not found: ID does not exist" Nov 02 23:02:11.332155 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:11.332107 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6"] Nov 02 23:02:11.343510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:11.341781 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-708f46a0-openshift-default-85b8966cfc-pmbr6"] Nov 02 23:02:11.822526 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:11.821976 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 02 23:02:11.822526 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:11.822048 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" podUID="d2cc2d36-def2-45fb-86bf-b35f56a969e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 02 23:02:12.792008 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:12.791969 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8eb89e7-8060-404a-b694-9d7bca211dac" path="/var/lib/kubelet/pods/d8eb89e7-8060-404a-b694-9d7bca211dac/volumes" Nov 02 23:02:12.825509 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:12.824908 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 02 23:02:12.825509 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:12.824970 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" podUID="d2cc2d36-def2-45fb-86bf-b35f56a969e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 02 23:02:13.821508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:13.820957 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 02 23:02:13.821508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:13.821054 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" podUID="d2cc2d36-def2-45fb-86bf-b35f56a969e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 02 23:02:14.969371 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:14.969317 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:02:14.969974 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:14.969755 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:02:14.970644 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:14.970617 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 02 23:02:17.194651 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:17.194609 5114 generic.go:356] "Generic (PLEG): container finished" podID="57299a9e-ed21-49ee-acdd-81831c2efd84" containerID="4aaca3c96642a52203751a35e36c120e36fcb7f70ecee0cbd6cc4e9433c4a40b" exitCode=0 Nov 02 23:02:17.195226 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:17.194689 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" event={"ID":"57299a9e-ed21-49ee-acdd-81831c2efd84","Type":"ContainerDied","Data":"4aaca3c96642a52203751a35e36c120e36fcb7f70ecee0cbd6cc4e9433c4a40b"} Nov 02 23:02:17.195226 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:17.195203 5114 scope.go:122] "RemoveContainer" containerID="4aaca3c96642a52203751a35e36c120e36fcb7f70ecee0cbd6cc4e9433c4a40b" Nov 02 23:02:17.196526 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:17.196501 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:02:17.196603 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:17.196572 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:02:17.196639 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:17.196604 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:02:18.201041 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:18.200991 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" event={"ID":"57299a9e-ed21-49ee-acdd-81831c2efd84","Type":"ContainerStarted","Data":"a47c38c0fa1246544943ebf531ba37b152705fc0c5c57950be5b80aa495b1aac"} Nov 02 23:02:18.559192 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:18.559161 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:02:18.560259 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:18.560237 5114 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 02 23:02:18.560333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:18.560313 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" podUID="57299a9e-ed21-49ee-acdd-81831c2efd84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 02 23:02:18.938599 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:18.938466 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:02:19.559644 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:19.559603 5114 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 02 23:02:19.560070 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:19.559673 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" podUID="57299a9e-ed21-49ee-acdd-81831c2efd84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 02 23:02:20.559585 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:20.559542 5114 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 02 23:02:20.559781 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:20.559609 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" podUID="57299a9e-ed21-49ee-acdd-81831c2efd84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 02 23:02:21.559324 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:21.559289 5114 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 02 23:02:21.559537 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:21.559342 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" podUID="57299a9e-ed21-49ee-acdd-81831c2efd84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 02 23:02:22.559627 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:22.559584 5114 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 02 23:02:22.560147 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:22.559655 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" podUID="57299a9e-ed21-49ee-acdd-81831c2efd84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 02 23:02:23.563630 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.563582 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:02:23.564880 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.564854 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:02:23.692299 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.692264 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx"] Nov 02 23:02:23.692716 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.692697 5114 cpu_manager.go:414] "containerMap: removing container" podUID="0dad888a-7ae1-481a-8f3d-877d6a1c52bf" containerName="istio-proxy" Nov 02 23:02:23.692814 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.692719 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dad888a-7ae1-481a-8f3d-877d6a1c52bf" containerName="istio-proxy" Nov 02 23:02:23.692814 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.692758 5114 cpu_manager.go:414] "containerMap: removing container" podUID="d8eb89e7-8060-404a-b694-9d7bca211dac" containerName="istio-proxy" Nov 02 23:02:23.692814 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.692766 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8eb89e7-8060-404a-b694-9d7bca211dac" containerName="istio-proxy" Nov 02 23:02:23.692814 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.692795 5114 cpu_manager.go:414] "containerMap: removing container" podUID="d8eb89e7-8060-404a-b694-9d7bca211dac" containerName="istio-proxy" Nov 02 23:02:23.692814 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.692802 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8eb89e7-8060-404a-b694-9d7bca211dac" containerName="istio-proxy" Nov 02 23:02:23.693066 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.692886 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8eb89e7-8060-404a-b694-9d7bca211dac" containerName="istio-proxy" Nov 02 23:02:23.694832 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.694806 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:02:23.747060 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.747019 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-08c2c6\"/\"openshift-service-ca.crt\"" Nov 02 23:02:23.782127 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.782093 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-08c2c6\"/\"kube-root-ca.crt\"" Nov 02 23:02:23.782541 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.782515 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-08c2c6\"/\"default-dockercfg-zg2jb\"" Nov 02 23:02:23.782682 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.782543 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-08c2c6\"/\"simulator-w2-08c2c6-kserve-self-signed-certs\"" Nov 02 23:02:23.783798 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.783776 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx"] Nov 02 23:02:23.793932 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.793894 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mffbw\" (UniqueName: \"kubernetes.io/projected/496c6815-1b56-4082-9ffd-9ea51a762a58-kube-api-access-mffbw\") pod \"simulator-w2-08c2c6-kserve-6b7769c645-bb5xx\" (UID: \"496c6815-1b56-4082-9ffd-9ea51a762a58\") " pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:02:23.794102 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.793952 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/496c6815-1b56-4082-9ffd-9ea51a762a58-tls-certs\") pod \"simulator-w2-08c2c6-kserve-6b7769c645-bb5xx\" (UID: \"496c6815-1b56-4082-9ffd-9ea51a762a58\") " pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:02:23.794102 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.794059 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-dshm\") pod \"simulator-w2-08c2c6-kserve-6b7769c645-bb5xx\" (UID: \"496c6815-1b56-4082-9ffd-9ea51a762a58\") " pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:02:23.794102 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.794095 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-tmp-dir\") pod \"simulator-w2-08c2c6-kserve-6b7769c645-bb5xx\" (UID: \"496c6815-1b56-4082-9ffd-9ea51a762a58\") " pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:02:23.794222 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.794112 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-home\") pod \"simulator-w2-08c2c6-kserve-6b7769c645-bb5xx\" (UID: \"496c6815-1b56-4082-9ffd-9ea51a762a58\") " pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:02:23.794222 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.794187 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-model-cache\") pod \"simulator-w2-08c2c6-kserve-6b7769c645-bb5xx\" (UID: \"496c6815-1b56-4082-9ffd-9ea51a762a58\") " pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:02:23.895602 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.895453 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mffbw\" (UniqueName: \"kubernetes.io/projected/496c6815-1b56-4082-9ffd-9ea51a762a58-kube-api-access-mffbw\") pod \"simulator-w2-08c2c6-kserve-6b7769c645-bb5xx\" (UID: \"496c6815-1b56-4082-9ffd-9ea51a762a58\") " pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:02:23.895602 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.895573 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/496c6815-1b56-4082-9ffd-9ea51a762a58-tls-certs\") pod \"simulator-w2-08c2c6-kserve-6b7769c645-bb5xx\" (UID: \"496c6815-1b56-4082-9ffd-9ea51a762a58\") " pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:02:23.895868 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.895668 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-dshm\") pod \"simulator-w2-08c2c6-kserve-6b7769c645-bb5xx\" (UID: \"496c6815-1b56-4082-9ffd-9ea51a762a58\") " pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:02:23.895868 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.895721 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-tmp-dir\") pod \"simulator-w2-08c2c6-kserve-6b7769c645-bb5xx\" (UID: \"496c6815-1b56-4082-9ffd-9ea51a762a58\") " pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:02:23.895868 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.895805 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-home\") pod \"simulator-w2-08c2c6-kserve-6b7769c645-bb5xx\" (UID: \"496c6815-1b56-4082-9ffd-9ea51a762a58\") " pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:02:23.895868 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.895840 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-model-cache\") pod \"simulator-w2-08c2c6-kserve-6b7769c645-bb5xx\" (UID: \"496c6815-1b56-4082-9ffd-9ea51a762a58\") " pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:02:23.896164 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.896140 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-home\") pod \"simulator-w2-08c2c6-kserve-6b7769c645-bb5xx\" (UID: \"496c6815-1b56-4082-9ffd-9ea51a762a58\") " pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:02:23.896268 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.896199 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-tmp-dir\") pod \"simulator-w2-08c2c6-kserve-6b7769c645-bb5xx\" (UID: \"496c6815-1b56-4082-9ffd-9ea51a762a58\") " pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:02:23.896268 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.896206 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-model-cache\") pod \"simulator-w2-08c2c6-kserve-6b7769c645-bb5xx\" (UID: \"496c6815-1b56-4082-9ffd-9ea51a762a58\") " pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:02:23.898128 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.898101 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-dshm\") pod \"simulator-w2-08c2c6-kserve-6b7769c645-bb5xx\" (UID: \"496c6815-1b56-4082-9ffd-9ea51a762a58\") " pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:02:23.898540 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.898519 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/496c6815-1b56-4082-9ffd-9ea51a762a58-tls-certs\") pod \"simulator-w2-08c2c6-kserve-6b7769c645-bb5xx\" (UID: \"496c6815-1b56-4082-9ffd-9ea51a762a58\") " pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:02:23.956303 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:23.956265 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mffbw\" (UniqueName: \"kubernetes.io/projected/496c6815-1b56-4082-9ffd-9ea51a762a58-kube-api-access-mffbw\") pod \"simulator-w2-08c2c6-kserve-6b7769c645-bb5xx\" (UID: \"496c6815-1b56-4082-9ffd-9ea51a762a58\") " pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:02:24.025040 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:24.024994 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:02:24.400074 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:24.399987 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx"] Nov 02 23:02:24.403129 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:02:24.403100 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod496c6815_1b56_4082_9ffd_9ea51a762a58.slice/crio-11b4dfb560228d52c743412d02fedfe2e88fa187bf2adab1aa598c18f3c6c0ee WatchSource:0}: Error finding container 11b4dfb560228d52c743412d02fedfe2e88fa187bf2adab1aa598c18f3c6c0ee: Status 404 returned error can't find the container with id 11b4dfb560228d52c743412d02fedfe2e88fa187bf2adab1aa598c18f3c6c0ee Nov 02 23:02:25.256876 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:25.256834 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" event={"ID":"496c6815-1b56-4082-9ffd-9ea51a762a58","Type":"ContainerStarted","Data":"7e6bcfa1b5cbae8308584bb89eacf2bb44f7ad371e9676be247515182ea4243a"} Nov 02 23:02:25.257286 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:25.256887 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" event={"ID":"496c6815-1b56-4082-9ffd-9ea51a762a58","Type":"ContainerStarted","Data":"11b4dfb560228d52c743412d02fedfe2e88fa187bf2adab1aa598c18f3c6c0ee"} Nov 02 23:02:25.374032 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:25.373970 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" podStartSLOduration=2.373953952 podStartE2EDuration="2.373953952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:25.372633003 +0000 UTC m=+820.725666401" watchObservedRunningTime="2026-11-02 23:02:25.373953952 +0000 UTC m=+820.726987324" Nov 02 23:02:26.025751 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:26.025717 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:02:34.025817 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:34.025756 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:02:34.039971 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:34.039936 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:02:34.312842 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:34.312803 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:02:37.583898 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:02:37.583851 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:02:47.928219 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:47.928178 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt"] Nov 02 23:02:47.928845 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:47.928823 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8eb89e7-8060-404a-b694-9d7bca211dac" containerName="istio-proxy" Nov 02 23:02:47.931506 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:47.931448 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:02:47.988327 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:47.988238 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ec2f9a\"/\"kube-root-ca.crt\"" Nov 02 23:02:48.014575 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:48.014538 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/592a8e63-9584-4890-b38e-900b0482f735-tls-certs\") pod \"simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt\" (UID: \"592a8e63-9584-4890-b38e-900b0482f735\") " pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:02:48.014787 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:48.014619 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-home\") pod \"simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt\" (UID: \"592a8e63-9584-4890-b38e-900b0482f735\") " pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:02:48.014833 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:48.014774 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-dshm\") pod \"simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt\" (UID: \"592a8e63-9584-4890-b38e-900b0482f735\") " pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:02:48.014833 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:48.014815 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-model-cache\") pod \"simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt\" (UID: \"592a8e63-9584-4890-b38e-900b0482f735\") " pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:02:48.014942 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:48.014919 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgf7x\" (UniqueName: \"kubernetes.io/projected/592a8e63-9584-4890-b38e-900b0482f735-kube-api-access-tgf7x\") pod \"simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt\" (UID: \"592a8e63-9584-4890-b38e-900b0482f735\") " pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:02:48.014985 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:48.014963 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-tmp-dir\") pod \"simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt\" (UID: \"592a8e63-9584-4890-b38e-900b0482f735\") " pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:02:48.033258 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:48.033217 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ec2f9a\"/\"openshift-service-ca.crt\"" Nov 02 23:02:48.033258 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:48.033221 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ec2f9a\"/\"default-dockercfg-c8pcq\"" Nov 02 23:02:48.033583 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:48.033288 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ec2f9a\"/\"simulator-w0-ec2f9a-kserve-self-signed-certs\"" Nov 02 23:02:48.034723 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:48.034698 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt"] Nov 02 23:02:48.116500 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:48.116454 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-home\") pod \"simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt\" (UID: \"592a8e63-9584-4890-b38e-900b0482f735\") " pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:02:48.116712 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:48.116533 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-dshm\") pod \"simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt\" (UID: \"592a8e63-9584-4890-b38e-900b0482f735\") " pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:02:48.116712 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:48.116550 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-model-cache\") pod \"simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt\" (UID: \"592a8e63-9584-4890-b38e-900b0482f735\") " pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:02:48.116712 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:48.116584 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgf7x\" (UniqueName: \"kubernetes.io/projected/592a8e63-9584-4890-b38e-900b0482f735-kube-api-access-tgf7x\") pod \"simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt\" (UID: \"592a8e63-9584-4890-b38e-900b0482f735\") " pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:02:48.116909 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:48.116741 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-tmp-dir\") pod \"simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt\" (UID: \"592a8e63-9584-4890-b38e-900b0482f735\") " pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:02:48.116909 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:48.116806 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/592a8e63-9584-4890-b38e-900b0482f735-tls-certs\") pod \"simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt\" (UID: \"592a8e63-9584-4890-b38e-900b0482f735\") " pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:02:48.117030 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:48.116957 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-home\") pod \"simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt\" (UID: \"592a8e63-9584-4890-b38e-900b0482f735\") " pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:02:48.117030 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:48.117005 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-model-cache\") pod \"simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt\" (UID: \"592a8e63-9584-4890-b38e-900b0482f735\") " pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:02:48.117212 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:48.117188 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-tmp-dir\") pod \"simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt\" (UID: \"592a8e63-9584-4890-b38e-900b0482f735\") " pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:02:48.119002 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:48.118973 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-dshm\") pod \"simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt\" (UID: \"592a8e63-9584-4890-b38e-900b0482f735\") " pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:02:48.119302 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:48.119282 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/592a8e63-9584-4890-b38e-900b0482f735-tls-certs\") pod \"simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt\" (UID: \"592a8e63-9584-4890-b38e-900b0482f735\") " pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:02:48.177793 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:48.177756 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgf7x\" (UniqueName: \"kubernetes.io/projected/592a8e63-9584-4890-b38e-900b0482f735-kube-api-access-tgf7x\") pod \"simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt\" (UID: \"592a8e63-9584-4890-b38e-900b0482f735\") " pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:02:48.243661 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:48.243542 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:02:48.623914 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:48.623875 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt"] Nov 02 23:02:48.628032 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:02:48.627989 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod592a8e63_9584_4890_b38e_900b0482f735.slice/crio-21c3c5d82fc7fa07a0de5053abd97b3c997273b2c3a9ea97ff829e80f98d4300 WatchSource:0}: Error finding container 21c3c5d82fc7fa07a0de5053abd97b3c997273b2c3a9ea97ff829e80f98d4300: Status 404 returned error can't find the container with id 21c3c5d82fc7fa07a0de5053abd97b3c997273b2c3a9ea97ff829e80f98d4300 Nov 02 23:02:49.372702 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:49.372665 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" event={"ID":"592a8e63-9584-4890-b38e-900b0482f735","Type":"ContainerStarted","Data":"b737a12e0c60ff2393934b3cbd6557792ad48996348aafaaf51a96e12b36a616"} Nov 02 23:02:49.372702 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:49.372706 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" event={"ID":"592a8e63-9584-4890-b38e-900b0482f735","Type":"ContainerStarted","Data":"21c3c5d82fc7fa07a0de5053abd97b3c997273b2c3a9ea97ff829e80f98d4300"} Nov 02 23:02:49.519991 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:49.519936 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" podStartSLOduration=2.519919045 podStartE2EDuration="2.519919045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:49.518359513 +0000 UTC m=+844.871392888" watchObservedRunningTime="2026-11-02 23:02:49.519919045 +0000 UTC m=+844.872952419" Nov 02 23:02:50.244529 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:50.244462 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:02:52.029086 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.029052 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf"] Nov 02 23:02:52.031374 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.031357 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.083848 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.083807 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b42983-openshift-default-dockercfg-xchbk\"" Nov 02 23:02:52.135386 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.135338 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf"] Nov 02 23:02:52.155729 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.155693 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-955v8\" (UniqueName: \"kubernetes.io/projected/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-kube-api-access-955v8\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.155933 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.155756 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.155933 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.155836 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.156053 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.155948 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.156053 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.155983 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.156053 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.156009 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.156148 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.156092 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.156183 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.156150 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.156216 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.156199 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.257430 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.257390 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.257652 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.257465 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-955v8\" (UniqueName: \"kubernetes.io/projected/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-kube-api-access-955v8\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.257652 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.257517 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.257652 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.257541 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.257652 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.257588 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.257652 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.257616 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.257652 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.257640 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.257967 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.257910 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.258054 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.258027 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.258121 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.258079 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.258195 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.258162 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.258251 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.258223 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.258600 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.258572 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.258705 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.258687 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.260641 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.260618 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.260857 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.260838 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.318658 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.318622 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.318807 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.318759 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-955v8\" (UniqueName: \"kubernetes.io/projected/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-kube-api-access-955v8\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.344277 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.344238 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:52.769472 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.769307 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf"] Nov 02 23:02:52.772336 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:02:52.772295 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafcd99fa_b983_4c6d_83a6_43cd089a8f5e.slice/crio-d31a6136abd0f0d1ee81ca2aff907fc9d923527f27a1d4d1a9bae7621e49e54d WatchSource:0}: Error finding container d31a6136abd0f0d1ee81ca2aff907fc9d923527f27a1d4d1a9bae7621e49e54d: Status 404 returned error can't find the container with id d31a6136abd0f0d1ee81ca2aff907fc9d923527f27a1d4d1a9bae7621e49e54d Nov 02 23:02:52.774808 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.774774 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:02:52.774880 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.774843 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:02:52.774880 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:52.774873 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:02:53.391840 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:53.391808 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" event={"ID":"afcd99fa-b983-4c6d-83a6-43cd089a8f5e","Type":"ContainerStarted","Data":"083de0e1b597425c6322de1b65f231e9af6f642b6efc2a64ac0115d0f8443f60"} Nov 02 23:02:53.392289 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:53.391859 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" event={"ID":"afcd99fa-b983-4c6d-83a6-43cd089a8f5e","Type":"ContainerStarted","Data":"d31a6136abd0f0d1ee81ca2aff907fc9d923527f27a1d4d1a9bae7621e49e54d"} Nov 02 23:02:53.531029 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:53.530945 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" podStartSLOduration=2.530919615 podStartE2EDuration="2.530919615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:53.529066696 +0000 UTC m=+848.882100069" watchObservedRunningTime="2026-11-02 23:02:53.530919615 +0000 UTC m=+848.883952992" Nov 02 23:02:54.345164 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:54.345124 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:54.346645 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:54.346621 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 23:02:54.346784 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:54.346673 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" podUID="afcd99fa-b983-4c6d-83a6-43cd089a8f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 23:02:55.345564 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:55.345529 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 23:02:55.345987 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:55.345582 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" podUID="afcd99fa-b983-4c6d-83a6-43cd089a8f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 23:02:56.345237 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:56.345203 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 23:02:56.345401 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:56.345266 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" podUID="afcd99fa-b983-4c6d-83a6-43cd089a8f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 23:02:57.344681 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.344644 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 23:02:57.345257 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.345224 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" podUID="afcd99fa-b983-4c6d-83a6-43cd089a8f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 23:02:57.348474 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.346680 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t"] Nov 02 23:02:57.349598 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.349571 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:02:57.405191 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.405151 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-home\") pod \"simulator-w3-96031a-kserve-787bf88b74-tmn4t\" (UID: \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\") " pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:02:57.435829 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.435594 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-tls-certs\") pod \"simulator-w3-96031a-kserve-787bf88b74-tmn4t\" (UID: \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\") " pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:02:57.435829 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.435669 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-tmp-dir\") pod \"simulator-w3-96031a-kserve-787bf88b74-tmn4t\" (UID: \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\") " pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:02:57.435829 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.435818 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-model-cache\") pod \"simulator-w3-96031a-kserve-787bf88b74-tmn4t\" (UID: \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\") " pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:02:57.436147 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.435902 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-dshm\") pod \"simulator-w3-96031a-kserve-787bf88b74-tmn4t\" (UID: \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\") " pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:02:57.436147 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.435949 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvrd4\" (UniqueName: \"kubernetes.io/projected/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-kube-api-access-lvrd4\") pod \"simulator-w3-96031a-kserve-787bf88b74-tmn4t\" (UID: \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\") " pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:02:57.452522 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.451282 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-96031a\"/\"openshift-service-ca.crt\"" Nov 02 23:02:57.452522 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.451315 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-96031a\"/\"kube-root-ca.crt\"" Nov 02 23:02:57.452786 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.452760 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t"] Nov 02 23:02:57.454149 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.454125 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-96031a\"/\"default-dockercfg-smtpz\"" Nov 02 23:02:57.501057 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.500889 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-96031a\"/\"simulator-w3-96031a-kserve-self-signed-certs\"" Nov 02 23:02:57.537132 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.537088 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-home\") pod \"simulator-w3-96031a-kserve-787bf88b74-tmn4t\" (UID: \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\") " pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:02:57.537323 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.537139 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-tls-certs\") pod \"simulator-w3-96031a-kserve-787bf88b74-tmn4t\" (UID: \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\") " pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:02:57.537323 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.537160 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-tmp-dir\") pod \"simulator-w3-96031a-kserve-787bf88b74-tmn4t\" (UID: \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\") " pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:02:57.537323 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.537208 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-model-cache\") pod \"simulator-w3-96031a-kserve-787bf88b74-tmn4t\" (UID: \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\") " pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:02:57.537323 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.537238 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-dshm\") pod \"simulator-w3-96031a-kserve-787bf88b74-tmn4t\" (UID: \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\") " pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:02:57.537323 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.537256 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvrd4\" (UniqueName: \"kubernetes.io/projected/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-kube-api-access-lvrd4\") pod \"simulator-w3-96031a-kserve-787bf88b74-tmn4t\" (UID: \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\") " pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:02:57.537929 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.537909 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-home\") pod \"simulator-w3-96031a-kserve-787bf88b74-tmn4t\" (UID: \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\") " pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:02:57.538519 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.538473 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-model-cache\") pod \"simulator-w3-96031a-kserve-787bf88b74-tmn4t\" (UID: \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\") " pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:02:57.538827 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.538805 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-tmp-dir\") pod \"simulator-w3-96031a-kserve-787bf88b74-tmn4t\" (UID: \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\") " pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:02:57.540509 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.540464 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-dshm\") pod \"simulator-w3-96031a-kserve-787bf88b74-tmn4t\" (UID: \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\") " pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:02:57.540664 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.540536 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-tls-certs\") pod \"simulator-w3-96031a-kserve-787bf88b74-tmn4t\" (UID: \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\") " pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:02:57.601636 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.601543 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvrd4\" (UniqueName: \"kubernetes.io/projected/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-kube-api-access-lvrd4\") pod \"simulator-w3-96031a-kserve-787bf88b74-tmn4t\" (UID: \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\") " pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:02:57.661786 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:57.661747 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:02:58.090813 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:58.090785 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t"] Nov 02 23:02:58.245841 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:58.245806 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:02:58.261116 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:58.261083 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:02:58.344878 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:58.344780 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 23:02:58.344878 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:58.344843 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" podUID="afcd99fa-b983-4c6d-83a6-43cd089a8f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 23:02:58.444445 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:58.444402 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" event={"ID":"fb16fc7d-3def-41dc-91a7-1103ccaf4c35","Type":"ContainerStarted","Data":"98de9b33e0b6bccb5280f8d48580c653c28c20f0f4e97c78969b4a852ab2af38"} Nov 02 23:02:58.444659 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:58.444456 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" event={"ID":"fb16fc7d-3def-41dc-91a7-1103ccaf4c35","Type":"ContainerStarted","Data":"6590208f65f9f023893795452d3f4023b70e4442012e777e31fdfe6ed74fc28b"} Nov 02 23:02:58.456468 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:58.456434 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:02:58.562475 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:58.561262 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" podStartSLOduration=1.561248137 podStartE2EDuration="1.561248137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:58.560244084 +0000 UTC m=+853.913277472" watchObservedRunningTime="2026-11-02 23:02:58.561248137 +0000 UTC m=+853.914281511" Nov 02 23:02:58.662671 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:58.662573 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:02:59.350081 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:59.350046 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:59.350623 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:59.350513 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:02:59.351356 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:02:59.351337 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:03:07.583904 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:03:07.583857 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:03:07.662069 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:07.662032 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:03:07.675147 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:07.675122 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:03:08.500956 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:08.500866 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:03:11.521235 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:11.521194 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr"] Nov 02 23:03:11.523578 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:11.523559 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:03:11.564078 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:11.564034 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-dshm\") pod \"premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr\" (UID: \"bcd6362b-2995-4ebc-972d-dfcee6e41357\") " pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:03:11.564254 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:11.564086 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-model-cache\") pod \"premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr\" (UID: \"bcd6362b-2995-4ebc-972d-dfcee6e41357\") " pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:03:11.564254 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:11.564191 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcd6362b-2995-4ebc-972d-dfcee6e41357-tls-certs\") pod \"premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr\" (UID: \"bcd6362b-2995-4ebc-972d-dfcee6e41357\") " pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:03:11.564363 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:11.564314 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-home\") pod \"premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr\" (UID: \"bcd6362b-2995-4ebc-972d-dfcee6e41357\") " pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:03:11.564363 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:11.564338 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-tmp-dir\") pod \"premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr\" (UID: \"bcd6362b-2995-4ebc-972d-dfcee6e41357\") " pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:03:11.564453 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:11.564377 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x79b8\" (UniqueName: \"kubernetes.io/projected/bcd6362b-2995-4ebc-972d-dfcee6e41357-kube-api-access-x79b8\") pod \"premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr\" (UID: \"bcd6362b-2995-4ebc-972d-dfcee6e41357\") " pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:03:11.625088 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:11.625048 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-08c2c6\"/\"premium-w2-08c2c6-kserve-self-signed-certs\"" Nov 02 23:03:11.626582 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:11.626557 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr"] Nov 02 23:03:11.664921 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:11.664877 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x79b8\" (UniqueName: \"kubernetes.io/projected/bcd6362b-2995-4ebc-972d-dfcee6e41357-kube-api-access-x79b8\") pod \"premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr\" (UID: \"bcd6362b-2995-4ebc-972d-dfcee6e41357\") " pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:03:11.665135 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:11.664942 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-dshm\") pod \"premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr\" (UID: \"bcd6362b-2995-4ebc-972d-dfcee6e41357\") " pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:03:11.665135 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:11.664965 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-model-cache\") pod \"premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr\" (UID: \"bcd6362b-2995-4ebc-972d-dfcee6e41357\") " pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:03:11.665135 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:11.664986 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcd6362b-2995-4ebc-972d-dfcee6e41357-tls-certs\") pod \"premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr\" (UID: \"bcd6362b-2995-4ebc-972d-dfcee6e41357\") " pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:03:11.665135 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:11.665080 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-home\") pod \"premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr\" (UID: \"bcd6362b-2995-4ebc-972d-dfcee6e41357\") " pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:03:11.665135 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:11.665098 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-tmp-dir\") pod \"premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr\" (UID: \"bcd6362b-2995-4ebc-972d-dfcee6e41357\") " pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:03:11.665581 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:11.665544 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-model-cache\") pod \"premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr\" (UID: \"bcd6362b-2995-4ebc-972d-dfcee6e41357\") " pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:03:11.665689 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:11.665555 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-tmp-dir\") pod \"premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr\" (UID: \"bcd6362b-2995-4ebc-972d-dfcee6e41357\") " pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:03:11.665689 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:11.665589 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-home\") pod \"premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr\" (UID: \"bcd6362b-2995-4ebc-972d-dfcee6e41357\") " pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:03:11.667667 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:11.667635 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-dshm\") pod \"premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr\" (UID: \"bcd6362b-2995-4ebc-972d-dfcee6e41357\") " pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:03:11.667878 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:11.667834 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcd6362b-2995-4ebc-972d-dfcee6e41357-tls-certs\") pod \"premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr\" (UID: \"bcd6362b-2995-4ebc-972d-dfcee6e41357\") " pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:03:11.726059 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:11.726011 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x79b8\" (UniqueName: \"kubernetes.io/projected/bcd6362b-2995-4ebc-972d-dfcee6e41357-kube-api-access-x79b8\") pod \"premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr\" (UID: \"bcd6362b-2995-4ebc-972d-dfcee6e41357\") " pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:03:11.835065 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:11.835022 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:03:12.229351 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:12.229318 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr"] Nov 02 23:03:12.231969 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:03:12.231935 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbcd6362b_2995_4ebc_972d_dfcee6e41357.slice/crio-3d60a6f1e0acda76cf9b0da163834b144b21c0d6c44c3751d335189de1ab5d17 WatchSource:0}: Error finding container 3d60a6f1e0acda76cf9b0da163834b144b21c0d6c44c3751d335189de1ab5d17: Status 404 returned error can't find the container with id 3d60a6f1e0acda76cf9b0da163834b144b21c0d6c44c3751d335189de1ab5d17 Nov 02 23:03:12.512440 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:12.512338 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" event={"ID":"bcd6362b-2995-4ebc-972d-dfcee6e41357","Type":"ContainerStarted","Data":"654794705cd2e4f9b0c5e8e9fff6a2f3e41755ed3bb9a16b69248aedef951f2f"} Nov 02 23:03:12.512440 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:12.512385 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" event={"ID":"bcd6362b-2995-4ebc-972d-dfcee6e41357","Type":"ContainerStarted","Data":"3d60a6f1e0acda76cf9b0da163834b144b21c0d6c44c3751d335189de1ab5d17"} Nov 02 23:03:12.629181 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:12.629122 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" podStartSLOduration=1.629104684 podStartE2EDuration="1.629104684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:12.62801524 +0000 UTC m=+867.981048618" watchObservedRunningTime="2026-11-02 23:03:12.629104684 +0000 UTC m=+867.982138059" Nov 02 23:03:12.835406 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:12.835365 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:03:21.628511 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.628451 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd"] Nov 02 23:03:21.630811 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.630793 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:03:21.661474 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.661434 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvxzf\" (UniqueName: \"kubernetes.io/projected/e9da41e7-2e4b-4912-9267-3f732f714155-kube-api-access-wvxzf\") pod \"simulator-w4-637fdb-kserve-566fc5cc95-xn8nd\" (UID: \"e9da41e7-2e4b-4912-9267-3f732f714155\") " pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:03:21.661676 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.661510 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-tmp-dir\") pod \"simulator-w4-637fdb-kserve-566fc5cc95-xn8nd\" (UID: \"e9da41e7-2e4b-4912-9267-3f732f714155\") " pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:03:21.661676 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.661618 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-model-cache\") pod \"simulator-w4-637fdb-kserve-566fc5cc95-xn8nd\" (UID: \"e9da41e7-2e4b-4912-9267-3f732f714155\") " pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:03:21.661765 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.661681 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9da41e7-2e4b-4912-9267-3f732f714155-tls-certs\") pod \"simulator-w4-637fdb-kserve-566fc5cc95-xn8nd\" (UID: \"e9da41e7-2e4b-4912-9267-3f732f714155\") " pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:03:21.661804 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.661761 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-dshm\") pod \"simulator-w4-637fdb-kserve-566fc5cc95-xn8nd\" (UID: \"e9da41e7-2e4b-4912-9267-3f732f714155\") " pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:03:21.661804 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.661801 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-home\") pod \"simulator-w4-637fdb-kserve-566fc5cc95-xn8nd\" (UID: \"e9da41e7-2e4b-4912-9267-3f732f714155\") " pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:03:21.732649 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.732618 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-637fdb\"/\"openshift-service-ca.crt\"" Nov 02 23:03:21.732649 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.732618 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-637fdb\"/\"kube-root-ca.crt\"" Nov 02 23:03:21.732875 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.732727 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-637fdb\"/\"simulator-w4-637fdb-kserve-self-signed-certs\"" Nov 02 23:03:21.732875 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.732753 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-637fdb\"/\"default-dockercfg-6ffdh\"" Nov 02 23:03:21.734333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.734310 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd"] Nov 02 23:03:21.762703 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.762660 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-model-cache\") pod \"simulator-w4-637fdb-kserve-566fc5cc95-xn8nd\" (UID: \"e9da41e7-2e4b-4912-9267-3f732f714155\") " pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:03:21.762891 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.762725 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9da41e7-2e4b-4912-9267-3f732f714155-tls-certs\") pod \"simulator-w4-637fdb-kserve-566fc5cc95-xn8nd\" (UID: \"e9da41e7-2e4b-4912-9267-3f732f714155\") " pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:03:21.762891 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.762756 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-dshm\") pod \"simulator-w4-637fdb-kserve-566fc5cc95-xn8nd\" (UID: \"e9da41e7-2e4b-4912-9267-3f732f714155\") " pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:03:21.762891 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.762788 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-home\") pod \"simulator-w4-637fdb-kserve-566fc5cc95-xn8nd\" (UID: \"e9da41e7-2e4b-4912-9267-3f732f714155\") " pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:03:21.762891 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.762832 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvxzf\" (UniqueName: \"kubernetes.io/projected/e9da41e7-2e4b-4912-9267-3f732f714155-kube-api-access-wvxzf\") pod \"simulator-w4-637fdb-kserve-566fc5cc95-xn8nd\" (UID: \"e9da41e7-2e4b-4912-9267-3f732f714155\") " pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:03:21.762891 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.762854 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-tmp-dir\") pod \"simulator-w4-637fdb-kserve-566fc5cc95-xn8nd\" (UID: \"e9da41e7-2e4b-4912-9267-3f732f714155\") " pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:03:21.763197 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.763163 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-model-cache\") pod \"simulator-w4-637fdb-kserve-566fc5cc95-xn8nd\" (UID: \"e9da41e7-2e4b-4912-9267-3f732f714155\") " pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:03:21.763268 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.763198 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-home\") pod \"simulator-w4-637fdb-kserve-566fc5cc95-xn8nd\" (UID: \"e9da41e7-2e4b-4912-9267-3f732f714155\") " pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:03:21.763321 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.763276 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-tmp-dir\") pod \"simulator-w4-637fdb-kserve-566fc5cc95-xn8nd\" (UID: \"e9da41e7-2e4b-4912-9267-3f732f714155\") " pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:03:21.765395 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.765353 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-dshm\") pod \"simulator-w4-637fdb-kserve-566fc5cc95-xn8nd\" (UID: \"e9da41e7-2e4b-4912-9267-3f732f714155\") " pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:03:21.765990 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.765967 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9da41e7-2e4b-4912-9267-3f732f714155-tls-certs\") pod \"simulator-w4-637fdb-kserve-566fc5cc95-xn8nd\" (UID: \"e9da41e7-2e4b-4912-9267-3f732f714155\") " pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:03:21.824255 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.824215 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvxzf\" (UniqueName: \"kubernetes.io/projected/e9da41e7-2e4b-4912-9267-3f732f714155-kube-api-access-wvxzf\") pod \"simulator-w4-637fdb-kserve-566fc5cc95-xn8nd\" (UID: \"e9da41e7-2e4b-4912-9267-3f732f714155\") " pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:03:21.836022 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.835985 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:03:21.849007 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.848979 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:03:21.942193 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:21.942095 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:03:22.256087 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.256001 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4"] Nov 02 23:03:22.258674 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.258653 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.292981 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.292940 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d"] Nov 02 23:03:22.295267 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.295245 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.320790 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.320753 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4"] Nov 02 23:03:22.368340 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.368316 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd"] Nov 02 23:03:22.371959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.371091 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.371959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.371159 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.371959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.371188 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.371959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.371235 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.371959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.371263 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95e2263e-c273-4505-baf4-65edea02ecd1-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.371959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.371288 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5fcr\" (UniqueName: \"kubernetes.io/projected/95e2263e-c273-4505-baf4-65edea02ecd1-kube-api-access-n5fcr\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.371959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.371346 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.371959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.371373 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95e2263e-c273-4505-baf4-65edea02ecd1-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.371959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.371399 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.371959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.371442 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.371959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.371521 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95e2263e-c273-4505-baf4-65edea02ecd1-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.371959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.371547 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.371959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.371578 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.371959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.371630 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.371959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.371657 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmjz6\" (UniqueName: \"kubernetes.io/projected/4df160f4-4ae0-4d2f-8464-5a7e0e792434-kube-api-access-fmjz6\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.371959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.371734 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.371959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.371762 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.371959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.371793 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.424579 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.424546 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d"] Nov 02 23:03:22.472890 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.472853 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95e2263e-c273-4505-baf4-65edea02ecd1-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.472890 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.472893 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.473149 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.472913 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.473149 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.472943 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.473149 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.472967 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmjz6\" (UniqueName: \"kubernetes.io/projected/4df160f4-4ae0-4d2f-8464-5a7e0e792434-kube-api-access-fmjz6\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.473149 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.473037 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.473460 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.473434 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.473533 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.473452 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.473533 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.473439 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.473626 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.473532 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.473626 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.473561 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.473626 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.473595 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.473626 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.473620 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.473829 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.473652 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.473829 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.473668 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.473829 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.473711 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.473829 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.473737 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95e2263e-c273-4505-baf4-65edea02ecd1-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.473829 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.473759 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5fcr\" (UniqueName: \"kubernetes.io/projected/95e2263e-c273-4505-baf4-65edea02ecd1-kube-api-access-n5fcr\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.473829 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.473816 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95e2263e-c273-4505-baf4-65edea02ecd1-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.474122 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.473889 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.474122 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.473901 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.474122 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.473819 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.474122 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.473959 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.474122 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.473997 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95e2263e-c273-4505-baf4-65edea02ecd1-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.474122 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.474024 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.474122 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.474070 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.474473 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.474134 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.474473 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.474407 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.476337 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.476310 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.476875 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.476853 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95e2263e-c273-4505-baf4-65edea02ecd1-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.477062 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.477039 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.477257 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.477217 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.533973 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.533939 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmjz6\" (UniqueName: \"kubernetes.io/projected/4df160f4-4ae0-4d2f-8464-5a7e0e792434-kube-api-access-fmjz6\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.534278 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.534253 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-lj67d\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.534318 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.534263 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95e2263e-c273-4505-baf4-65edea02ecd1-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.534353 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.534320 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5fcr\" (UniqueName: \"kubernetes.io/projected/95e2263e-c273-4505-baf4-65edea02ecd1-kube-api-access-n5fcr\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.561175 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.561134 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" event={"ID":"e9da41e7-2e4b-4912-9267-3f732f714155","Type":"ContainerStarted","Data":"f8db1932a558d1205aa11438a5d67b6a0756db6d8680b5fc33ef1c571456e2bd"} Nov 02 23:03:22.561395 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.561190 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" event={"ID":"e9da41e7-2e4b-4912-9267-3f732f714155","Type":"ContainerStarted","Data":"daf2d4e34c9bf34e7b3700d64000d0de535af3503356357a4411eb1cc26a10d5"} Nov 02 23:03:22.572619 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.572587 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:22.572761 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.572722 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:03:22.608150 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.608112 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:22.678931 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.678884 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" podStartSLOduration=1.6788663000000001 podStartE2EDuration="1.6788663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:22.676589865 +0000 UTC m=+878.029623242" watchObservedRunningTime="2026-11-02 23:03:22.6788663 +0000 UTC m=+878.031899684" Nov 02 23:03:22.943090 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.943003 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:03:22.948897 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.948877 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4"] Nov 02 23:03:22.951753 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:03:22.951719 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e2263e_c273_4505_baf4_65edea02ecd1.slice/crio-cc29a41285525f4add8cff4a33cddc293ba478a8b64f83eb6dd6c7dbdc2fd629 WatchSource:0}: Error finding container cc29a41285525f4add8cff4a33cddc293ba478a8b64f83eb6dd6c7dbdc2fd629: Status 404 returned error can't find the container with id cc29a41285525f4add8cff4a33cddc293ba478a8b64f83eb6dd6c7dbdc2fd629 Nov 02 23:03:22.954527 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.954469 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:03:22.954611 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.954565 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:03:22.954611 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.954593 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:03:22.998759 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:22.998735 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d"] Nov 02 23:03:23.001813 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:03:23.001782 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df160f4_4ae0_4d2f_8464_5a7e0e792434.slice/crio-df2559d4fa7cf8b372d8e59c1c0df6393fcd989b3b7c9216e7d8a54e83a2ba16 WatchSource:0}: Error finding container df2559d4fa7cf8b372d8e59c1c0df6393fcd989b3b7c9216e7d8a54e83a2ba16: Status 404 returned error can't find the container with id df2559d4fa7cf8b372d8e59c1c0df6393fcd989b3b7c9216e7d8a54e83a2ba16 Nov 02 23:03:23.004433 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:23.004392 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:03:23.004616 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:23.004472 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:03:23.004616 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:23.004526 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:03:23.568735 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:23.568699 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" event={"ID":"4df160f4-4ae0-4d2f-8464-5a7e0e792434","Type":"ContainerStarted","Data":"9d418505c24e6abab1e4b7a6c5ff84eb0374134b67844c3b8fc60b8833f96465"} Nov 02 23:03:23.568944 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:23.568765 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" event={"ID":"4df160f4-4ae0-4d2f-8464-5a7e0e792434","Type":"ContainerStarted","Data":"df2559d4fa7cf8b372d8e59c1c0df6393fcd989b3b7c9216e7d8a54e83a2ba16"} Nov 02 23:03:23.570465 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:23.570438 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" event={"ID":"95e2263e-c273-4505-baf4-65edea02ecd1","Type":"ContainerStarted","Data":"3cfd00ed5b081a10f19b01b68046d16ccc9d45085cdcca6f78ff81bea5f09684"} Nov 02 23:03:23.570630 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:23.570511 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" event={"ID":"95e2263e-c273-4505-baf4-65edea02ecd1","Type":"ContainerStarted","Data":"cc29a41285525f4add8cff4a33cddc293ba478a8b64f83eb6dd6c7dbdc2fd629"} Nov 02 23:03:23.573003 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:23.572983 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:23.627512 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:23.626591 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:23.723510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:23.714107 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" podStartSLOduration=1.7140877749999999 podStartE2EDuration="1.714087775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:23.701459526 +0000 UTC m=+879.054492901" watchObservedRunningTime="2026-11-02 23:03:23.714087775 +0000 UTC m=+879.067121151" Nov 02 23:03:23.828777 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:23.828644 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" podStartSLOduration=1.8286235450000001 podStartE2EDuration="1.828623545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:23.826057369 +0000 UTC m=+879.179090745" watchObservedRunningTime="2026-11-02 23:03:23.828623545 +0000 UTC m=+879.181656921" Nov 02 23:03:24.574612 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:24.574568 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 02 23:03:24.574791 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:24.574638 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" podUID="95e2263e-c273-4505-baf4-65edea02ecd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 02 23:03:24.615182 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:24.615140 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 02 23:03:24.619612 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:24.619546 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" podUID="4df160f4-4ae0-4d2f-8464-5a7e0e792434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 02 23:03:25.573410 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:25.573367 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 02 23:03:25.573906 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:25.573426 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" podUID="95e2263e-c273-4505-baf4-65edea02ecd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 02 23:03:25.617972 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:25.617928 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 02 23:03:25.618238 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:25.618207 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" podUID="4df160f4-4ae0-4d2f-8464-5a7e0e792434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 02 23:03:26.572968 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:26.572924 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 02 23:03:26.573180 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:26.572980 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" podUID="95e2263e-c273-4505-baf4-65edea02ecd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 02 23:03:26.617510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:26.615896 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 02 23:03:26.617510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:26.615963 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" podUID="4df160f4-4ae0-4d2f-8464-5a7e0e792434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 02 23:03:27.572961 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:27.572923 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 02 23:03:27.573165 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:27.572988 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" podUID="95e2263e-c273-4505-baf4-65edea02ecd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 02 23:03:27.631508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:27.631182 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 02 23:03:27.631508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:27.631245 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" podUID="4df160f4-4ae0-4d2f-8464-5a7e0e792434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 02 23:03:28.573911 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:28.573879 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 02 23:03:28.574102 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:28.573935 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" podUID="95e2263e-c273-4505-baf4-65edea02ecd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 02 23:03:28.622511 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:28.609917 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 02 23:03:28.622511 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:28.609978 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" podUID="4df160f4-4ae0-4d2f-8464-5a7e0e792434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 02 23:03:29.577393 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:29.577360 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:29.577935 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:29.577776 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:29.578617 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:29.578598 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:03:29.613007 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:29.612977 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:29.613288 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:29.613269 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:29.613990 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:29.613972 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:03:31.942700 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:31.942657 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:03:31.962344 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:31.962319 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:03:32.623709 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:32.623674 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:03:37.585598 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:03:37.585555 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:03:39.252592 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:39.252547 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv"] Nov 02 23:03:39.255992 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:39.255972 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:03:39.297228 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:39.297187 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-model-cache\") pod \"premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv\" (UID: \"72301e09-ed03-4bc9-8264-028de7930a48\") " pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:03:39.297411 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:39.297239 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72301e09-ed03-4bc9-8264-028de7930a48-tls-certs\") pod \"premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv\" (UID: \"72301e09-ed03-4bc9-8264-028de7930a48\") " pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:03:39.297411 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:39.297309 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvcbk\" (UniqueName: \"kubernetes.io/projected/72301e09-ed03-4bc9-8264-028de7930a48-kube-api-access-wvcbk\") pod \"premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv\" (UID: \"72301e09-ed03-4bc9-8264-028de7930a48\") " pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:03:39.297411 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:39.297342 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-home\") pod \"premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv\" (UID: \"72301e09-ed03-4bc9-8264-028de7930a48\") " pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:03:39.297568 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:39.297473 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-dshm\") pod \"premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv\" (UID: \"72301e09-ed03-4bc9-8264-028de7930a48\") " pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:03:39.297605 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:39.297569 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-tmp-dir\") pod \"premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv\" (UID: \"72301e09-ed03-4bc9-8264-028de7930a48\") " pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:03:39.357243 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:39.357209 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ec2f9a\"/\"premium-w0-ec2f9a-kserve-self-signed-certs\"" Nov 02 23:03:39.359268 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:39.359239 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv"] Nov 02 23:03:39.398825 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:39.398791 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-model-cache\") pod \"premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv\" (UID: \"72301e09-ed03-4bc9-8264-028de7930a48\") " pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:03:39.398825 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:39.398831 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72301e09-ed03-4bc9-8264-028de7930a48-tls-certs\") pod \"premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv\" (UID: \"72301e09-ed03-4bc9-8264-028de7930a48\") " pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:03:39.399085 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:39.398852 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvcbk\" (UniqueName: \"kubernetes.io/projected/72301e09-ed03-4bc9-8264-028de7930a48-kube-api-access-wvcbk\") pod \"premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv\" (UID: \"72301e09-ed03-4bc9-8264-028de7930a48\") " pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:03:39.399085 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:39.398885 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-home\") pod \"premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv\" (UID: \"72301e09-ed03-4bc9-8264-028de7930a48\") " pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:03:39.399085 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:39.398950 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-dshm\") pod \"premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv\" (UID: \"72301e09-ed03-4bc9-8264-028de7930a48\") " pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:03:39.399085 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:39.398981 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-tmp-dir\") pod \"premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv\" (UID: \"72301e09-ed03-4bc9-8264-028de7930a48\") " pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:03:39.399294 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:39.399269 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-model-cache\") pod \"premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv\" (UID: \"72301e09-ed03-4bc9-8264-028de7930a48\") " pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:03:39.399294 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:39.399278 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-home\") pod \"premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv\" (UID: \"72301e09-ed03-4bc9-8264-028de7930a48\") " pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:03:39.399396 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:39.399368 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-tmp-dir\") pod \"premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv\" (UID: \"72301e09-ed03-4bc9-8264-028de7930a48\") " pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:03:39.401503 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:39.401457 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-dshm\") pod \"premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv\" (UID: \"72301e09-ed03-4bc9-8264-028de7930a48\") " pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:03:39.401789 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:39.401767 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72301e09-ed03-4bc9-8264-028de7930a48-tls-certs\") pod \"premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv\" (UID: \"72301e09-ed03-4bc9-8264-028de7930a48\") " pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:03:39.460323 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:39.460285 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvcbk\" (UniqueName: \"kubernetes.io/projected/72301e09-ed03-4bc9-8264-028de7930a48-kube-api-access-wvcbk\") pod \"premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv\" (UID: \"72301e09-ed03-4bc9-8264-028de7930a48\") " pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:03:39.568013 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:39.567969 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:03:39.969532 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:39.969502 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv"] Nov 02 23:03:39.971086 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:03:39.971063 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72301e09_ed03_4bc9_8264_028de7930a48.slice/crio-edde84b9657d9b6ba3b11dd16d346f50da2ab88a58c723efb0c0c7f3729cc273 WatchSource:0}: Error finding container edde84b9657d9b6ba3b11dd16d346f50da2ab88a58c723efb0c0c7f3729cc273: Status 404 returned error can't find the container with id edde84b9657d9b6ba3b11dd16d346f50da2ab88a58c723efb0c0c7f3729cc273 Nov 02 23:03:40.665166 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:40.665122 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" event={"ID":"72301e09-ed03-4bc9-8264-028de7930a48","Type":"ContainerStarted","Data":"b7ff04c47fefdab73a5e1a07e45869138da57d7832b4ee3c42eef1ed97b24e2e"} Nov 02 23:03:40.665630 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:40.665176 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" event={"ID":"72301e09-ed03-4bc9-8264-028de7930a48","Type":"ContainerStarted","Data":"edde84b9657d9b6ba3b11dd16d346f50da2ab88a58c723efb0c0c7f3729cc273"} Nov 02 23:03:40.781988 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:40.781928 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" podStartSLOduration=1.781913328 podStartE2EDuration="1.781913328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:40.781276934 +0000 UTC m=+896.134310310" watchObservedRunningTime="2026-11-02 23:03:40.781913328 +0000 UTC m=+896.134946703" Nov 02 23:03:41.568540 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:41.568506 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:03:49.568333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:49.568300 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:03:49.580962 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:49.580931 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:03:49.720560 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:49.720529 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:03:53.396459 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:53.396418 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6"] Nov 02 23:03:53.401417 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:53.401390 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 02 23:03:53.457233 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:53.457201 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b42983\"/\"default-dockercfg-qt2p2\"" Nov 02 23:03:53.507123 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:53.507094 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b42983\"/\"kube-root-ca.crt\"" Nov 02 23:03:53.507123 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:53.507094 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b42983\"/\"openshift-service-ca.crt\"" Nov 02 23:03:53.508601 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:53.508580 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6"] Nov 02 23:03:53.556726 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:53.556616 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z8xv\" (UniqueName: \"kubernetes.io/projected/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-kube-api-access-5z8xv\") pod \"simulator-w1-b42983-kserve-f659c5658-4cpq6\" (UID: \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\") " pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 02 23:03:53.556968 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:53.556769 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-model-cache\") pod \"simulator-w1-b42983-kserve-f659c5658-4cpq6\" (UID: \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\") " pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 02 23:03:53.556968 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:53.556856 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-tls-certs\") pod \"simulator-w1-b42983-kserve-f659c5658-4cpq6\" (UID: \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\") " pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 02 23:03:53.556968 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:53.556893 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-dshm\") pod \"simulator-w1-b42983-kserve-f659c5658-4cpq6\" (UID: \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\") " pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 02 23:03:53.557117 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:53.557082 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-home\") pod \"simulator-w1-b42983-kserve-f659c5658-4cpq6\" (UID: \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\") " pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 02 23:03:53.557186 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:53.557120 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-tmp-dir\") pod \"simulator-w1-b42983-kserve-f659c5658-4cpq6\" (UID: \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\") " pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 02 23:03:53.557469 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:53.557453 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b42983\"/\"simulator-w1-b42983-kserve-self-signed-certs\"" Nov 02 23:03:53.658428 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:53.658325 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-model-cache\") pod \"simulator-w1-b42983-kserve-f659c5658-4cpq6\" (UID: \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\") " pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 02 23:03:53.658658 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:53.658429 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-tls-certs\") pod \"simulator-w1-b42983-kserve-f659c5658-4cpq6\" (UID: \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\") " pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 02 23:03:53.658658 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:53.658469 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-dshm\") pod \"simulator-w1-b42983-kserve-f659c5658-4cpq6\" (UID: \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\") " pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 02 23:03:53.658658 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:53.658580 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-home\") pod \"simulator-w1-b42983-kserve-f659c5658-4cpq6\" (UID: \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\") " pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 02 23:03:53.658658 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:53.658601 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-tmp-dir\") pod \"simulator-w1-b42983-kserve-f659c5658-4cpq6\" (UID: \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\") " pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 02 23:03:53.658874 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:53.658669 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z8xv\" (UniqueName: \"kubernetes.io/projected/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-kube-api-access-5z8xv\") pod \"simulator-w1-b42983-kserve-f659c5658-4cpq6\" (UID: \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\") " pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 02 23:03:53.658874 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:53.658814 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-model-cache\") pod \"simulator-w1-b42983-kserve-f659c5658-4cpq6\" (UID: \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\") " pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 02 23:03:53.659007 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:53.658982 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-home\") pod \"simulator-w1-b42983-kserve-f659c5658-4cpq6\" (UID: \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\") " pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 02 23:03:53.659059 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:53.659034 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-tmp-dir\") pod \"simulator-w1-b42983-kserve-f659c5658-4cpq6\" (UID: \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\") " pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 02 23:03:53.661261 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:53.661235 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-dshm\") pod \"simulator-w1-b42983-kserve-f659c5658-4cpq6\" (UID: \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\") " pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 02 23:03:53.661630 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:53.661606 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-tls-certs\") pod \"simulator-w1-b42983-kserve-f659c5658-4cpq6\" (UID: \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\") " pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 02 23:03:53.720926 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:53.720882 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z8xv\" (UniqueName: \"kubernetes.io/projected/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-kube-api-access-5z8xv\") pod \"simulator-w1-b42983-kserve-f659c5658-4cpq6\" (UID: \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\") " pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 02 23:03:54.014175 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:54.014079 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 02 23:03:54.393154 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:54.393126 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6"] Nov 02 23:03:54.395931 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:03:54.395901 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b13b7b2_8504_4eb5_864e_5b6eb22cd7b6.slice/crio-22b41d5b4130e29e972d5e7c662a5a77ddec0ecf30f08ef455a51e798972cb36 WatchSource:0}: Error finding container 22b41d5b4130e29e972d5e7c662a5a77ddec0ecf30f08ef455a51e798972cb36: Status 404 returned error can't find the container with id 22b41d5b4130e29e972d5e7c662a5a77ddec0ecf30f08ef455a51e798972cb36 Nov 02 23:03:54.739163 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:54.739052 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" event={"ID":"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6","Type":"ContainerStarted","Data":"195950387f07e660041bc231096baa0853186416d1fe4c1760d21c2c02134969"} Nov 02 23:03:54.739163 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:54.739113 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" event={"ID":"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6","Type":"ContainerStarted","Data":"22b41d5b4130e29e972d5e7c662a5a77ddec0ecf30f08ef455a51e798972cb36"} Nov 02 23:03:54.855121 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:54.855062 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" podStartSLOduration=1.855040112 podStartE2EDuration="1.855040112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:54.854617601 +0000 UTC m=+910.207650980" watchObservedRunningTime="2026-11-02 23:03:54.855040112 +0000 UTC m=+910.208073486" Nov 02 23:03:55.015021 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:03:55.014926 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 02 23:04:04.015156 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:04:04.015111 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 02 23:04:04.027950 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:04:04.027918 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 02 23:04:04.803770 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:04:04.803739 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 02 23:04:07.584387 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:04:07.584350 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:04:37.584832 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:04:37.584795 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:05:02.798014 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:02.797979 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh"] Nov 02 23:05:02.802009 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:02.801980 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:02.854068 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:02.854030 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-637fdb-openshift-default-dockercfg-brzsm\"" Nov 02 23:05:02.900734 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:02.900688 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:02.900921 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:02.900831 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:02.900921 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:02.900878 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsttz\" (UniqueName: \"kubernetes.io/projected/13beeff1-dd93-473c-b8c8-55ff28a8d746-kube-api-access-qsttz\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:02.900921 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:02.900906 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:02.901078 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:02.900925 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:02.901115 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:02.901072 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:02.901115 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:02.901109 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:02.901255 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:02.901231 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13beeff1-dd93-473c-b8c8-55ff28a8d746-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:02.901309 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:02.901273 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:02.905552 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:02.905476 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh"] Nov 02 23:05:03.001963 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:03.001922 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:03.002129 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:03.002005 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:03.002129 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:03.002059 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13beeff1-dd93-473c-b8c8-55ff28a8d746-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:03.002129 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:03.002076 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:03.002129 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:03.002119 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:03.002323 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:03.002178 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:03.002323 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:03.002209 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsttz\" (UniqueName: \"kubernetes.io/projected/13beeff1-dd93-473c-b8c8-55ff28a8d746-kube-api-access-qsttz\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:03.002323 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:03.002239 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:03.002323 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:03.002264 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:03.002562 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:03.002442 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:03.002826 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:03.002791 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:03.002929 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:03.002829 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:03.002929 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:03.002893 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:03.003052 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:03.003032 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13beeff1-dd93-473c-b8c8-55ff28a8d746-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:03.004524 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:03.004502 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:03.004794 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:03.004774 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:03.063475 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:03.063405 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:03.063475 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:03.063437 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsttz\" (UniqueName: \"kubernetes.io/projected/13beeff1-dd93-473c-b8c8-55ff28a8d746-kube-api-access-qsttz\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:03.116884 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:03.116838 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:03.546579 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:03.546552 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh"] Nov 02 23:05:03.548702 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:05:03.548677 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13beeff1_dd93_473c_b8c8_55ff28a8d746.slice/crio-c736aa104ff741d1ebbce7b61f6adbdae6755da938df51e620022c5fd46dc097 WatchSource:0}: Error finding container c736aa104ff741d1ebbce7b61f6adbdae6755da938df51e620022c5fd46dc097: Status 404 returned error can't find the container with id c736aa104ff741d1ebbce7b61f6adbdae6755da938df51e620022c5fd46dc097 Nov 02 23:05:03.551129 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:03.551079 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:05:03.551258 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:03.551191 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:05:03.551258 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:03.551237 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:05:04.096401 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:04.096362 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" event={"ID":"13beeff1-dd93-473c-b8c8-55ff28a8d746","Type":"ContainerStarted","Data":"1fd295d2b5085b486f32f44dc83899206c1a24d768d84a6013d0374793e112be"} Nov 02 23:05:04.096866 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:04.096440 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" event={"ID":"13beeff1-dd93-473c-b8c8-55ff28a8d746","Type":"ContainerStarted","Data":"c736aa104ff741d1ebbce7b61f6adbdae6755da938df51e620022c5fd46dc097"} Nov 02 23:05:04.117956 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:04.117912 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:04.119431 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:04.119407 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 02 23:05:04.119582 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:04.119451 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" podUID="13beeff1-dd93-473c-b8c8-55ff28a8d746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 02 23:05:04.218347 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:04.218275 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" podStartSLOduration=2.218251832 podStartE2EDuration="2.218251832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:05:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:05:04.216235307 +0000 UTC m=+979.569268683" watchObservedRunningTime="2026-11-02 23:05:04.218251832 +0000 UTC m=+979.571285208" Nov 02 23:05:05.126512 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:05.121633 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 02 23:05:05.126512 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:05.121714 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" podUID="13beeff1-dd93-473c-b8c8-55ff28a8d746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 02 23:05:06.127510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:06.121925 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 02 23:05:06.127510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:06.121987 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" podUID="13beeff1-dd93-473c-b8c8-55ff28a8d746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 02 23:05:07.126509 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:07.125948 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 02 23:05:07.126509 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:07.126013 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" podUID="13beeff1-dd93-473c-b8c8-55ff28a8d746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 02 23:05:07.584500 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:05:07.584437 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:05:08.121979 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:08.121934 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 02 23:05:08.122281 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:08.122249 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" podUID="13beeff1-dd93-473c-b8c8-55ff28a8d746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 02 23:05:09.124887 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:09.124848 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 02 23:05:09.125536 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:09.125499 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" podUID="13beeff1-dd93-473c-b8c8-55ff28a8d746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 02 23:05:10.117885 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:10.117842 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 02 23:05:10.118072 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:10.117927 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" podUID="13beeff1-dd93-473c-b8c8-55ff28a8d746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 02 23:05:11.121880 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:11.121847 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:11.122283 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:11.122129 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:11.122818 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:11.122798 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:05:33.269591 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.268838 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c"] Nov 02 23:05:33.293522 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.293013 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.338813 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.336025 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c"] Nov 02 23:05:33.382249 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.382221 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ec2f9a-openshift-default-dockercfg-4nt5j\"" Nov 02 23:05:33.465115 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.465071 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxkgb\" (UniqueName: \"kubernetes.io/projected/45a38607-6428-49c6-ab92-eb788f47a236-kube-api-access-fxkgb\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.465305 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.465143 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.465305 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.465218 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45a38607-6428-49c6-ab92-eb788f47a236-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.465305 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.465243 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a38607-6428-49c6-ab92-eb788f47a236-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.465305 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.465273 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.465305 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.465298 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.465567 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.465539 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.465619 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.465587 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a38607-6428-49c6-ab92-eb788f47a236-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.465732 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.465706 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.566758 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.566693 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45a38607-6428-49c6-ab92-eb788f47a236-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.566758 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.566749 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a38607-6428-49c6-ab92-eb788f47a236-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.567096 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.566800 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.567096 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.566834 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.567096 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.566966 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.567096 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.567002 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a38607-6428-49c6-ab92-eb788f47a236-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.567096 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.567060 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.567425 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.567111 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxkgb\" (UniqueName: \"kubernetes.io/projected/45a38607-6428-49c6-ab92-eb788f47a236-kube-api-access-fxkgb\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.567425 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.567169 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.567716 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.567684 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.568055 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.568026 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.568532 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.568509 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.569126 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.568672 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.569126 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.569055 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a38607-6428-49c6-ab92-eb788f47a236-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.573013 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.572988 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a38607-6428-49c6-ab92-eb788f47a236-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.577806 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.577766 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.628434 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.628363 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45a38607-6428-49c6-ab92-eb788f47a236-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.629960 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.629930 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxkgb\" (UniqueName: \"kubernetes.io/projected/45a38607-6428-49c6-ab92-eb788f47a236-kube-api-access-fxkgb\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:33.915030 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:33.914923 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:34.358402 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:34.358372 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c"] Nov 02 23:05:34.361855 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:05:34.361643 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a38607_6428_49c6_ab92_eb788f47a236.slice/crio-1cece0912f2bba46826120e8ec59410d4c57d2bfb2c638bd63be7288235af570 WatchSource:0}: Error finding container 1cece0912f2bba46826120e8ec59410d4c57d2bfb2c638bd63be7288235af570: Status 404 returned error can't find the container with id 1cece0912f2bba46826120e8ec59410d4c57d2bfb2c638bd63be7288235af570 Nov 02 23:05:34.364802 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:34.364757 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:05:34.364944 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:34.364855 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:05:34.364944 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:34.364899 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:05:35.290584 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:35.290546 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" event={"ID":"45a38607-6428-49c6-ab92-eb788f47a236","Type":"ContainerStarted","Data":"d07ec33e90efc5e6926b5f4687266b352ce875d0da2c9d0cb59f326966e6b4e6"} Nov 02 23:05:35.290745 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:35.290597 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" event={"ID":"45a38607-6428-49c6-ab92-eb788f47a236","Type":"ContainerStarted","Data":"1cece0912f2bba46826120e8ec59410d4c57d2bfb2c638bd63be7288235af570"} Nov 02 23:05:35.524512 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:35.520825 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" podStartSLOduration=2.5208021130000002 podStartE2EDuration="2.520802113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:05:35.462652205 +0000 UTC m=+1010.815685582" watchObservedRunningTime="2026-11-02 23:05:35.520802113 +0000 UTC m=+1010.873835490" Nov 02 23:05:35.921512 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:35.915591 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:35.921512 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:35.917839 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 23:05:35.921512 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:35.917923 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" podUID="45a38607-6428-49c6-ab92-eb788f47a236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 23:05:36.927510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:36.926960 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 23:05:36.927510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:36.927028 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" podUID="45a38607-6428-49c6-ab92-eb788f47a236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 23:05:37.584838 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:05:37.584794 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:05:37.919572 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:37.918947 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 23:05:37.919572 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:37.919012 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" podUID="45a38607-6428-49c6-ab92-eb788f47a236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 23:05:38.927511 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:38.926959 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 23:05:38.927511 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:38.927028 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" podUID="45a38607-6428-49c6-ab92-eb788f47a236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 23:05:39.929511 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:39.922976 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 23:05:39.929511 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:39.923046 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" podUID="45a38607-6428-49c6-ab92-eb788f47a236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 23:05:40.916123 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:40.916082 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 23:05:40.916343 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:40.916138 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" podUID="45a38607-6428-49c6-ab92-eb788f47a236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 23:05:41.919884 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:41.919853 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:41.920646 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:41.920141 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:41.920950 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:41.920932 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:05:56.164207 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.164169 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv"] Nov 02 23:05:56.168013 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.167982 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.214015 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.213974 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm"] Nov 02 23:05:56.217639 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.217595 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.255418 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.255380 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151a9d61-9d15-403d-955a-c398c7bec160-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.255418 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.255431 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-887kk\" (UniqueName: \"kubernetes.io/projected/151a9d61-9d15-403d-955a-c398c7bec160-kube-api-access-887kk\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.255694 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.255646 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.255762 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.255734 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.255809 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.255790 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151a9d61-9d15-403d-955a-c398c7bec160-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.255896 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.255879 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.255943 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.255914 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.256090 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.256072 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.256208 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.256187 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/151a9d61-9d15-403d-955a-c398c7bec160-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.302988 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.302945 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv"] Nov 02 23:05:56.357042 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.356992 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/151a9d61-9d15-403d-955a-c398c7bec160-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.357264 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.357060 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151a9d61-9d15-403d-955a-c398c7bec160-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.357264 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.357102 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887kk\" (UniqueName: \"kubernetes.io/projected/151a9d61-9d15-403d-955a-c398c7bec160-kube-api-access-887kk\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.357264 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.357159 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.357264 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.357191 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.357264 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.357224 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.371053 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.370965 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151a9d61-9d15-403d-955a-c398c7bec160-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.374818 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.374778 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.374990 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.374852 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151a9d61-9d15-403d-955a-c398c7bec160-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.374990 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.374887 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.374990 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.374951 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.375206 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.374991 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.375206 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.375035 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.375206 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.375102 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.375206 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.375110 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.375391 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.375253 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.375541 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.375520 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151a9d61-9d15-403d-955a-c398c7bec160-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.375632 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.375611 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.375767 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.375639 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrl5h\" (UniqueName: \"kubernetes.io/projected/d05d07a8-a317-42d7-b48c-7be5d5fed71a-kube-api-access-jrl5h\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.375767 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.375672 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.375895 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.375819 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.375895 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.375867 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.375976 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.375931 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.376166 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.376149 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.378128 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.378100 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.378255 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.378226 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm"] Nov 02 23:05:56.418742 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.418655 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/151a9d61-9d15-403d-955a-c398c7bec160-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.419232 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.419206 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-887kk\" (UniqueName: \"kubernetes.io/projected/151a9d61-9d15-403d-955a-c398c7bec160-kube-api-access-887kk\") pod \"maas-default-gateway-openshift-default-694db55c87-hlxtv\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.477565 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.477525 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.477565 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.477570 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.477860 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.477612 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.477860 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.477648 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.477860 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.477670 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.477860 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.477688 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrl5h\" (UniqueName: \"kubernetes.io/projected/d05d07a8-a317-42d7-b48c-7be5d5fed71a-kube-api-access-jrl5h\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.477860 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.477726 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.477860 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.477763 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.477860 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.477819 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.478218 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.478136 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.478218 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.478203 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.478332 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.478220 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.478523 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.478497 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.479107 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.479083 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.480419 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.480388 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.480419 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.480408 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.481409 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.481391 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:56.540272 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.540194 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.540426 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.540399 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrl5h\" (UniqueName: \"kubernetes.io/projected/d05d07a8-a317-42d7-b48c-7be5d5fed71a-kube-api-access-jrl5h\") pod \"maas-default-gateway-openshift-default-694db55c87-2d2hm\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.832110 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.832061 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:56.926538 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.926468 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv"] Nov 02 23:05:56.935240 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:05:56.935202 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151a9d61_9d15_403d_955a_c398c7bec160.slice/crio-58f7457ec21b550be03c6f600c1f9650cbafbaa5f6f12eac4a699497e680194b WatchSource:0}: Error finding container 58f7457ec21b550be03c6f600c1f9650cbafbaa5f6f12eac4a699497e680194b: Status 404 returned error can't find the container with id 58f7457ec21b550be03c6f600c1f9650cbafbaa5f6f12eac4a699497e680194b Nov 02 23:05:56.937953 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.937909 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:05:56.938052 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.938007 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:05:56.938110 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:56.938063 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:05:57.230839 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:57.230458 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm"] Nov 02 23:05:57.241393 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:05:57.241349 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05d07a8_a317_42d7_b48c_7be5d5fed71a.slice/crio-d448b60da2a0e1cff16e3a2ae9fab8a81c26d8a9dcbd3b2a477f3997274f50c1 WatchSource:0}: Error finding container d448b60da2a0e1cff16e3a2ae9fab8a81c26d8a9dcbd3b2a477f3997274f50c1: Status 404 returned error can't find the container with id d448b60da2a0e1cff16e3a2ae9fab8a81c26d8a9dcbd3b2a477f3997274f50c1 Nov 02 23:05:57.246937 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:57.246890 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:05:57.247069 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:57.246992 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:05:57.247069 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:57.247038 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:05:57.437264 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:57.436157 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" event={"ID":"151a9d61-9d15-403d-955a-c398c7bec160","Type":"ContainerStarted","Data":"0a11f6b86f5dcf550af8206d7741a883839e4c212d8a52319aeac931d22f70a1"} Nov 02 23:05:57.437264 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:57.436228 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" event={"ID":"151a9d61-9d15-403d-955a-c398c7bec160","Type":"ContainerStarted","Data":"58f7457ec21b550be03c6f600c1f9650cbafbaa5f6f12eac4a699497e680194b"} Nov 02 23:05:57.440382 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:57.440345 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" event={"ID":"d05d07a8-a317-42d7-b48c-7be5d5fed71a","Type":"ContainerStarted","Data":"dbf7f411da1fdb0a2cc651c98cdf266e1c274e2b70d6fb156ec9f336a551d90b"} Nov 02 23:05:57.440810 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:57.440761 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" event={"ID":"d05d07a8-a317-42d7-b48c-7be5d5fed71a","Type":"ContainerStarted","Data":"d448b60da2a0e1cff16e3a2ae9fab8a81c26d8a9dcbd3b2a477f3997274f50c1"} Nov 02 23:05:57.482230 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:57.482125 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:05:57.558403 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:57.558328 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" podStartSLOduration=1.558310149 podStartE2EDuration="1.558310149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:05:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:05:57.556871631 +0000 UTC m=+1032.909905019" watchObservedRunningTime="2026-11-02 23:05:57.558310149 +0000 UTC m=+1032.911343525" Nov 02 23:05:57.683036 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:57.682972 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" podStartSLOduration=1.682949555 podStartE2EDuration="1.682949555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:05:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:05:57.680798247 +0000 UTC m=+1033.033831625" watchObservedRunningTime="2026-11-02 23:05:57.682949555 +0000 UTC m=+1033.035982931" Nov 02 23:05:57.832782 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:57.832742 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:05:58.483166 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:58.483128 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-hlxtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 23:05:58.483663 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:58.483222 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" podUID="151a9d61-9d15-403d-955a-c398c7bec160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 23:05:58.841510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:58.841410 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-2d2hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 23:05:58.841710 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:58.841508 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" podUID="d05d07a8-a317-42d7-b48c-7be5d5fed71a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 23:05:59.484210 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:59.484159 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-hlxtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 23:05:59.484772 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:59.484434 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" podUID="151a9d61-9d15-403d-955a-c398c7bec160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 23:05:59.838510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:59.837915 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-2d2hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 23:05:59.838510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:05:59.837993 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" podUID="d05d07a8-a317-42d7-b48c-7be5d5fed71a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 23:06:00.481938 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:00.481903 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-hlxtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 23:06:00.482224 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:00.482196 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" podUID="151a9d61-9d15-403d-955a-c398c7bec160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 23:06:00.856519 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:00.855679 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-2d2hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 23:06:00.856519 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:00.855750 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" podUID="d05d07a8-a317-42d7-b48c-7be5d5fed71a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 23:06:01.482221 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:01.482184 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-hlxtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 23:06:01.482420 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:01.482236 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" podUID="151a9d61-9d15-403d-955a-c398c7bec160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 23:06:01.839509 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:01.838790 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-2d2hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 23:06:01.839509 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:01.838855 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" podUID="d05d07a8-a317-42d7-b48c-7be5d5fed71a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 23:06:02.489505 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:02.484867 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-hlxtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 23:06:02.489505 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:02.484927 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" podUID="151a9d61-9d15-403d-955a-c398c7bec160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 23:06:02.834959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:02.834917 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-2d2hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 23:06:02.835263 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:02.835226 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" podUID="d05d07a8-a317-42d7-b48c-7be5d5fed71a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 23:06:03.482371 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:03.482330 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-hlxtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 23:06:03.482608 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:03.482401 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" podUID="151a9d61-9d15-403d-955a-c398c7bec160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 23:06:03.833249 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:03.833198 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-2d2hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 23:06:03.833724 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:03.833280 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" podUID="d05d07a8-a317-42d7-b48c-7be5d5fed71a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 23:06:04.482155 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:04.482101 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-hlxtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 23:06:04.482384 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:04.482174 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" podUID="151a9d61-9d15-403d-955a-c398c7bec160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 23:06:04.833536 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:04.833468 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-2d2hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 23:06:04.833952 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:04.833562 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" podUID="d05d07a8-a317-42d7-b48c-7be5d5fed71a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 23:06:06.166621 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:06.166583 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:06:06.168082 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:06.168049 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:06:06.169616 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:06.169589 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:06:06.448994 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:06.448902 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:06:06.449669 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:06.449629 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:06:06.455980 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:06.455320 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:06:07.595265 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:06:07.595232 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:06:37.584935 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:06:37.584897 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:06:46.834612 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:46.834574 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr"] Nov 02 23:06:46.838594 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:46.838567 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:06:46.905967 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:46.905926 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wwdl\" (UniqueName: \"kubernetes.io/projected/5efc1b0f-5388-4059-b8da-5ae6c7a48501-kube-api-access-2wwdl\") pod \"distinct-w1-b42983-kserve-7f96779c74-72hmr\" (UID: \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:06:46.905967 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:46.905969 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-home\") pod \"distinct-w1-b42983-kserve-7f96779c74-72hmr\" (UID: \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:06:46.906212 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:46.906007 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-dshm\") pod \"distinct-w1-b42983-kserve-7f96779c74-72hmr\" (UID: \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:06:46.906212 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:46.906185 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-model-cache\") pod \"distinct-w1-b42983-kserve-7f96779c74-72hmr\" (UID: \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:06:46.906306 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:46.906288 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5efc1b0f-5388-4059-b8da-5ae6c7a48501-tls-certs\") pod \"distinct-w1-b42983-kserve-7f96779c74-72hmr\" (UID: \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:06:46.906384 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:46.906368 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-tmp-dir\") pod \"distinct-w1-b42983-kserve-7f96779c74-72hmr\" (UID: \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:06:46.940174 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:46.940131 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b42983\"/\"distinct-w1-b42983-kserve-self-signed-certs\"" Nov 02 23:06:46.942505 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:46.942460 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr"] Nov 02 23:06:47.007081 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:47.007037 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-dshm\") pod \"distinct-w1-b42983-kserve-7f96779c74-72hmr\" (UID: \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:06:47.007297 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:47.007111 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-model-cache\") pod \"distinct-w1-b42983-kserve-7f96779c74-72hmr\" (UID: \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:06:47.007297 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:47.007206 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5efc1b0f-5388-4059-b8da-5ae6c7a48501-tls-certs\") pod \"distinct-w1-b42983-kserve-7f96779c74-72hmr\" (UID: \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:06:47.007297 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:47.007280 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-tmp-dir\") pod \"distinct-w1-b42983-kserve-7f96779c74-72hmr\" (UID: \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:06:47.007535 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:47.007339 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wwdl\" (UniqueName: \"kubernetes.io/projected/5efc1b0f-5388-4059-b8da-5ae6c7a48501-kube-api-access-2wwdl\") pod \"distinct-w1-b42983-kserve-7f96779c74-72hmr\" (UID: \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:06:47.007535 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:47.007363 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-home\") pod \"distinct-w1-b42983-kserve-7f96779c74-72hmr\" (UID: \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:06:47.007890 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:47.007863 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-home\") pod \"distinct-w1-b42983-kserve-7f96779c74-72hmr\" (UID: \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:06:47.008821 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:47.008790 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-tmp-dir\") pod \"distinct-w1-b42983-kserve-7f96779c74-72hmr\" (UID: \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:06:47.009304 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:47.009270 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-model-cache\") pod \"distinct-w1-b42983-kserve-7f96779c74-72hmr\" (UID: \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:06:47.010520 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:47.010496 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-dshm\") pod \"distinct-w1-b42983-kserve-7f96779c74-72hmr\" (UID: \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:06:47.011181 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:47.011157 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5efc1b0f-5388-4059-b8da-5ae6c7a48501-tls-certs\") pod \"distinct-w1-b42983-kserve-7f96779c74-72hmr\" (UID: \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:06:47.069169 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:47.069130 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wwdl\" (UniqueName: \"kubernetes.io/projected/5efc1b0f-5388-4059-b8da-5ae6c7a48501-kube-api-access-2wwdl\") pod \"distinct-w1-b42983-kserve-7f96779c74-72hmr\" (UID: \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\") " pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:06:47.150769 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:47.150675 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:06:47.565414 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:47.565383 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr"] Nov 02 23:06:47.566169 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:06:47.566145 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5efc1b0f_5388_4059_b8da_5ae6c7a48501.slice/crio-b0e0981e60ba2c8e2cf2e4424e2109ef56fc1358ec5ccc1bfd8f38be757a8108 WatchSource:0}: Error finding container b0e0981e60ba2c8e2cf2e4424e2109ef56fc1358ec5ccc1bfd8f38be757a8108: Status 404 returned error can't find the container with id b0e0981e60ba2c8e2cf2e4424e2109ef56fc1358ec5ccc1bfd8f38be757a8108 Nov 02 23:06:47.748193 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:47.748146 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" event={"ID":"5efc1b0f-5388-4059-b8da-5ae6c7a48501","Type":"ContainerStarted","Data":"80662916c5e185f6e8096c5b78dcb677582a379f82735a7ae45c98199a866452"} Nov 02 23:06:47.748380 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:47.748206 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" event={"ID":"5efc1b0f-5388-4059-b8da-5ae6c7a48501","Type":"ContainerStarted","Data":"b0e0981e60ba2c8e2cf2e4424e2109ef56fc1358ec5ccc1bfd8f38be757a8108"} Nov 02 23:06:47.865722 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:47.865613 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" podStartSLOduration=1.865597086 podStartE2EDuration="1.865597086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:06:47.864360499 +0000 UTC m=+1083.217393874" watchObservedRunningTime="2026-11-02 23:06:47.865597086 +0000 UTC m=+1083.218630519" Nov 02 23:06:48.151769 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:48.151672 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:06:57.151731 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:57.151686 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:06:57.165159 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:57.165128 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:06:57.814510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:57.814455 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:06:58.748255 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:58.748212 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd"] Nov 02 23:06:58.753795 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:58.753762 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:06:58.856519 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:58.830116 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd"] Nov 02 23:06:58.856893 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:58.855710 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-637fdb\"/\"distinct-w4-637fdb-kserve-self-signed-certs\"" Nov 02 23:06:58.975510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:58.972628 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgz4n\" (UniqueName: \"kubernetes.io/projected/4d705cd8-8b43-4229-9df0-b0980e7728aa-kube-api-access-pgz4n\") pod \"distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd\" (UID: \"4d705cd8-8b43-4229-9df0-b0980e7728aa\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:06:58.975510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:58.972771 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-model-cache\") pod \"distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd\" (UID: \"4d705cd8-8b43-4229-9df0-b0980e7728aa\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:06:58.975510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:58.972830 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-dshm\") pod \"distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd\" (UID: \"4d705cd8-8b43-4229-9df0-b0980e7728aa\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:06:58.975510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:58.972940 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d705cd8-8b43-4229-9df0-b0980e7728aa-tls-certs\") pod \"distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd\" (UID: \"4d705cd8-8b43-4229-9df0-b0980e7728aa\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:06:58.975510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:58.973048 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-home\") pod \"distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd\" (UID: \"4d705cd8-8b43-4229-9df0-b0980e7728aa\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:06:58.975510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:58.973112 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-tmp-dir\") pod \"distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd\" (UID: \"4d705cd8-8b43-4229-9df0-b0980e7728aa\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:06:59.077316 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:59.077268 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-model-cache\") pod \"distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd\" (UID: \"4d705cd8-8b43-4229-9df0-b0980e7728aa\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:06:59.082749 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:59.082711 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-dshm\") pod \"distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd\" (UID: \"4d705cd8-8b43-4229-9df0-b0980e7728aa\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:06:59.083053 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:59.083034 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d705cd8-8b43-4229-9df0-b0980e7728aa-tls-certs\") pod \"distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd\" (UID: \"4d705cd8-8b43-4229-9df0-b0980e7728aa\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:06:59.083275 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:59.083256 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-home\") pod \"distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd\" (UID: \"4d705cd8-8b43-4229-9df0-b0980e7728aa\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:06:59.083447 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:59.083420 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-tmp-dir\") pod \"distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd\" (UID: \"4d705cd8-8b43-4229-9df0-b0980e7728aa\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:06:59.087732 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:59.087700 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgz4n\" (UniqueName: \"kubernetes.io/projected/4d705cd8-8b43-4229-9df0-b0980e7728aa-kube-api-access-pgz4n\") pod \"distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd\" (UID: \"4d705cd8-8b43-4229-9df0-b0980e7728aa\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:06:59.088971 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:59.088946 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-model-cache\") pod \"distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd\" (UID: \"4d705cd8-8b43-4229-9df0-b0980e7728aa\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:06:59.096509 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:59.089857 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-home\") pod \"distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd\" (UID: \"4d705cd8-8b43-4229-9df0-b0980e7728aa\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:06:59.096509 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:59.090141 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-tmp-dir\") pod \"distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd\" (UID: \"4d705cd8-8b43-4229-9df0-b0980e7728aa\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:06:59.118051 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:59.118004 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d705cd8-8b43-4229-9df0-b0980e7728aa-tls-certs\") pod \"distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd\" (UID: \"4d705cd8-8b43-4229-9df0-b0980e7728aa\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:06:59.122294 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:59.122252 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-dshm\") pod \"distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd\" (UID: \"4d705cd8-8b43-4229-9df0-b0980e7728aa\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:06:59.168674 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:59.168637 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgz4n\" (UniqueName: \"kubernetes.io/projected/4d705cd8-8b43-4229-9df0-b0980e7728aa-kube-api-access-pgz4n\") pod \"distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd\" (UID: \"4d705cd8-8b43-4229-9df0-b0980e7728aa\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:06:59.394416 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:59.394306 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:06:59.776041 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:59.776015 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd"] Nov 02 23:06:59.778283 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:06:59.778249 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d705cd8_8b43_4229_9df0_b0980e7728aa.slice/crio-7308c25b56562a8b9d25c2e1884d5a408242f90ca71e22de8db7fbf2c934b6ba WatchSource:0}: Error finding container 7308c25b56562a8b9d25c2e1884d5a408242f90ca71e22de8db7fbf2c934b6ba: Status 404 returned error can't find the container with id 7308c25b56562a8b9d25c2e1884d5a408242f90ca71e22de8db7fbf2c934b6ba Nov 02 23:06:59.813620 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:06:59.813592 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" event={"ID":"4d705cd8-8b43-4229-9df0-b0980e7728aa","Type":"ContainerStarted","Data":"7308c25b56562a8b9d25c2e1884d5a408242f90ca71e22de8db7fbf2c934b6ba"} Nov 02 23:07:00.820258 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:00.820216 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" event={"ID":"4d705cd8-8b43-4229-9df0-b0980e7728aa","Type":"ContainerStarted","Data":"2b93fcca28e10fac7c49321432d4b01ce25724d8e84b35f560df0d477df0f727"} Nov 02 23:07:00.936799 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:00.936747 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" podStartSLOduration=2.936733925 podStartE2EDuration="2.936733925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:07:00.936346866 +0000 UTC m=+1096.289380242" watchObservedRunningTime="2026-11-02 23:07:00.936733925 +0000 UTC m=+1096.289767299" Nov 02 23:07:01.394515 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:01.394461 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:07:07.585780 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:07:07.585732 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:07:09.394860 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:09.394820 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:07:09.407931 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:09.407902 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:07:09.879122 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:09.879084 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:07:37.586901 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:07:37.586853 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:07:49.876763 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:49.876726 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd"] Nov 02 23:07:49.881050 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:49.881024 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:07:49.893800 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:49.893765 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f850c9fa-e500-4772-8454-a5172e446770-tls-certs\") pod \"distinct-w3-96031a-kserve-6c69b88c98-lmshd\" (UID: \"f850c9fa-e500-4772-8454-a5172e446770\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:07:49.893962 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:49.893807 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-tmp-dir\") pod \"distinct-w3-96031a-kserve-6c69b88c98-lmshd\" (UID: \"f850c9fa-e500-4772-8454-a5172e446770\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:07:49.893962 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:49.893905 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-home\") pod \"distinct-w3-96031a-kserve-6c69b88c98-lmshd\" (UID: \"f850c9fa-e500-4772-8454-a5172e446770\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:07:49.893962 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:49.893931 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-dshm\") pod \"distinct-w3-96031a-kserve-6c69b88c98-lmshd\" (UID: \"f850c9fa-e500-4772-8454-a5172e446770\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:07:49.894096 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:49.894078 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npv27\" (UniqueName: \"kubernetes.io/projected/f850c9fa-e500-4772-8454-a5172e446770-kube-api-access-npv27\") pod \"distinct-w3-96031a-kserve-6c69b88c98-lmshd\" (UID: \"f850c9fa-e500-4772-8454-a5172e446770\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:07:49.894148 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:49.894115 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-model-cache\") pod \"distinct-w3-96031a-kserve-6c69b88c98-lmshd\" (UID: \"f850c9fa-e500-4772-8454-a5172e446770\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:07:49.939130 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:49.939092 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd"] Nov 02 23:07:49.985616 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:49.985577 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-96031a\"/\"distinct-w3-96031a-kserve-self-signed-certs\"" Nov 02 23:07:49.995563 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:49.995516 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f850c9fa-e500-4772-8454-a5172e446770-tls-certs\") pod \"distinct-w3-96031a-kserve-6c69b88c98-lmshd\" (UID: \"f850c9fa-e500-4772-8454-a5172e446770\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:07:49.995563 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:49.995562 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-tmp-dir\") pod \"distinct-w3-96031a-kserve-6c69b88c98-lmshd\" (UID: \"f850c9fa-e500-4772-8454-a5172e446770\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:07:49.995801 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:49.995590 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-home\") pod \"distinct-w3-96031a-kserve-6c69b88c98-lmshd\" (UID: \"f850c9fa-e500-4772-8454-a5172e446770\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:07:49.995801 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:49.995605 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-dshm\") pod \"distinct-w3-96031a-kserve-6c69b88c98-lmshd\" (UID: \"f850c9fa-e500-4772-8454-a5172e446770\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:07:49.995801 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:49.995684 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npv27\" (UniqueName: \"kubernetes.io/projected/f850c9fa-e500-4772-8454-a5172e446770-kube-api-access-npv27\") pod \"distinct-w3-96031a-kserve-6c69b88c98-lmshd\" (UID: \"f850c9fa-e500-4772-8454-a5172e446770\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:07:49.995801 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:49.995711 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-model-cache\") pod \"distinct-w3-96031a-kserve-6c69b88c98-lmshd\" (UID: \"f850c9fa-e500-4772-8454-a5172e446770\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:07:49.996114 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:49.996072 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-tmp-dir\") pod \"distinct-w3-96031a-kserve-6c69b88c98-lmshd\" (UID: \"f850c9fa-e500-4772-8454-a5172e446770\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:07:49.996114 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:49.996106 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-home\") pod \"distinct-w3-96031a-kserve-6c69b88c98-lmshd\" (UID: \"f850c9fa-e500-4772-8454-a5172e446770\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:07:49.996246 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:49.996145 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-model-cache\") pod \"distinct-w3-96031a-kserve-6c69b88c98-lmshd\" (UID: \"f850c9fa-e500-4772-8454-a5172e446770\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:07:49.998675 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:49.998647 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-dshm\") pod \"distinct-w3-96031a-kserve-6c69b88c98-lmshd\" (UID: \"f850c9fa-e500-4772-8454-a5172e446770\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:07:49.999008 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:49.998990 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f850c9fa-e500-4772-8454-a5172e446770-tls-certs\") pod \"distinct-w3-96031a-kserve-6c69b88c98-lmshd\" (UID: \"f850c9fa-e500-4772-8454-a5172e446770\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:07:50.057761 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:50.057726 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npv27\" (UniqueName: \"kubernetes.io/projected/f850c9fa-e500-4772-8454-a5172e446770-kube-api-access-npv27\") pod \"distinct-w3-96031a-kserve-6c69b88c98-lmshd\" (UID: \"f850c9fa-e500-4772-8454-a5172e446770\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:07:50.194249 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:50.194139 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:07:50.637878 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:50.637852 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd"] Nov 02 23:07:50.639553 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:07:50.639518 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf850c9fa_e500_4772_8454_a5172e446770.slice/crio-6a512ac8aec386162951c15273e5687c7e4916153a027ab37d260beb40266fcd WatchSource:0}: Error finding container 6a512ac8aec386162951c15273e5687c7e4916153a027ab37d260beb40266fcd: Status 404 returned error can't find the container with id 6a512ac8aec386162951c15273e5687c7e4916153a027ab37d260beb40266fcd Nov 02 23:07:51.103217 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:51.103175 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" event={"ID":"f850c9fa-e500-4772-8454-a5172e446770","Type":"ContainerStarted","Data":"2b54f75a1e0f0551c482460ae98ea70b93e7565573be0b69f756a7e4ffcc03b2"} Nov 02 23:07:51.103707 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:51.103227 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" event={"ID":"f850c9fa-e500-4772-8454-a5172e446770","Type":"ContainerStarted","Data":"6a512ac8aec386162951c15273e5687c7e4916153a027ab37d260beb40266fcd"} Nov 02 23:07:51.194729 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:51.194701 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:07:51.219992 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:07:51.219927 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" podStartSLOduration=2.219911014 podStartE2EDuration="2.219911014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:07:51.218993551 +0000 UTC m=+1146.572026927" watchObservedRunningTime="2026-11-02 23:07:51.219911014 +0000 UTC m=+1146.572944390" Nov 02 23:08:00.194348 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:00.194305 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:08:00.208205 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:00.208172 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:08:01.164995 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:01.164962 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:08:02.722210 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.722166 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws"] Nov 02 23:08:02.725954 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.725936 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:02.829760 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.829722 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws"] Nov 02 23:08:02.845043 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.845000 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:02.845043 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.845048 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:02.845293 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.845083 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:02.845388 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.845349 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:02.845451 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.845395 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:02.845605 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.845523 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:02.845605 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.845558 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:02.845605 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.845588 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pchb\" (UniqueName: \"kubernetes.io/projected/774684c7-0c28-43c7-bba7-d2c1bcccad16-kube-api-access-2pchb\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:02.845776 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.845676 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774684c7-0c28-43c7-bba7-d2c1bcccad16-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:02.946544 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.946477 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:02.946743 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.946553 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:02.946743 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.946593 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:02.946743 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.946716 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:02.946943 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.946743 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:02.946943 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.946803 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:02.946943 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.946835 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:02.946943 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.946864 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pchb\" (UniqueName: \"kubernetes.io/projected/774684c7-0c28-43c7-bba7-d2c1bcccad16-kube-api-access-2pchb\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:02.946943 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.946894 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774684c7-0c28-43c7-bba7-d2c1bcccad16-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:02.947224 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.947055 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:02.947224 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.947198 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:02.947383 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.947358 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:02.947383 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.947373 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:02.947723 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.947705 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774684c7-0c28-43c7-bba7-d2c1bcccad16-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:02.949526 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.949504 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:02.949630 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:02.949600 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:03.007611 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:03.007535 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pchb\" (UniqueName: \"kubernetes.io/projected/774684c7-0c28-43c7-bba7-d2c1bcccad16-kube-api-access-2pchb\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:03.007611 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:03.007532 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:03.040019 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:03.039986 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:03.470535 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:03.470508 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws"] Nov 02 23:08:03.472651 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:08:03.472620 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774684c7_0c28_43c7_bba7_d2c1bcccad16.slice/crio-1fa365df914a7da1d274511b102567ea242481fbe06faad4a44b9818c2dd9d71 WatchSource:0}: Error finding container 1fa365df914a7da1d274511b102567ea242481fbe06faad4a44b9818c2dd9d71: Status 404 returned error can't find the container with id 1fa365df914a7da1d274511b102567ea242481fbe06faad4a44b9818c2dd9d71 Nov 02 23:08:03.475025 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:03.474993 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:08:03.475092 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:03.475064 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:08:03.475126 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:03.475110 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:08:04.171080 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:04.171044 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" event={"ID":"774684c7-0c28-43c7-bba7-d2c1bcccad16","Type":"ContainerStarted","Data":"a1dff3951ca35c7200893e7aff7631d395972abde1cf7d99c47aeb4ec0df36bd"} Nov 02 23:08:04.171549 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:04.171105 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" event={"ID":"774684c7-0c28-43c7-bba7-d2c1bcccad16","Type":"ContainerStarted","Data":"1fa365df914a7da1d274511b102567ea242481fbe06faad4a44b9818c2dd9d71"} Nov 02 23:08:04.404426 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:04.404360 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" podStartSLOduration=2.404342394 podStartE2EDuration="2.404342394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:08:04.353687767 +0000 UTC m=+1159.706721142" watchObservedRunningTime="2026-11-02 23:08:04.404342394 +0000 UTC m=+1159.757375768" Nov 02 23:08:05.040399 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:05.040357 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:05.042108 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:05.042079 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 23:08:05.042180 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:05.042137 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" podUID="774684c7-0c28-43c7-bba7-d2c1bcccad16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 23:08:06.041281 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:06.041237 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 23:08:06.041856 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:06.041298 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" podUID="774684c7-0c28-43c7-bba7-d2c1bcccad16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 23:08:07.040810 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:07.040767 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 23:08:07.041010 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:07.040847 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" podUID="774684c7-0c28-43c7-bba7-d2c1bcccad16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 23:08:07.583972 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:08:07.583933 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:08:08.040955 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:08.040917 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 23:08:08.041182 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:08.040977 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" podUID="774684c7-0c28-43c7-bba7-d2c1bcccad16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 23:08:09.040616 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:09.040574 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 23:08:09.041063 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:09.040633 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" podUID="774684c7-0c28-43c7-bba7-d2c1bcccad16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 23:08:10.041032 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:10.040983 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 23:08:10.041448 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:10.041072 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" podUID="774684c7-0c28-43c7-bba7-d2c1bcccad16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 23:08:11.045508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:11.045456 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:11.045997 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:11.045854 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:11.046566 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:11.046544 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:08:26.601169 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:26.601067 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" podUID="bcd6362b-2995-4ebc-972d-dfcee6e41357" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 23:08:26.659959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:26.659911 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" podUID="e9da41e7-2e4b-4912-9267-3f732f714155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 23:08:32.042030 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:32.041994 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8"] Nov 02 23:08:32.045713 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:32.045695 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:08:32.146969 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:32.146935 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-637fdb\"/\"distinct-2-w4-637fdb-kserve-self-signed-certs\"" Nov 02 23:08:32.149143 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:32.149116 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8"] Nov 02 23:08:32.170541 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:32.170468 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-tmp-dir\") pod \"distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8\" (UID: \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:08:32.170541 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:32.170534 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmflk\" (UniqueName: \"kubernetes.io/projected/5e3275e5-f49f-4ca7-8d49-03178a77fee6-kube-api-access-mmflk\") pod \"distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8\" (UID: \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:08:32.170757 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:32.170638 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-home\") pod \"distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8\" (UID: \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:08:32.170795 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:32.170749 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-dshm\") pod \"distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8\" (UID: \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:08:32.170829 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:32.170800 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e3275e5-f49f-4ca7-8d49-03178a77fee6-tls-certs\") pod \"distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8\" (UID: \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:08:32.170862 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:32.170851 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-model-cache\") pod \"distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8\" (UID: \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:08:32.271956 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:32.271920 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-home\") pod \"distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8\" (UID: \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:08:32.272181 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:32.272006 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-dshm\") pod \"distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8\" (UID: \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:08:32.272181 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:32.272055 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e3275e5-f49f-4ca7-8d49-03178a77fee6-tls-certs\") pod \"distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8\" (UID: \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:08:32.272181 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:32.272108 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-model-cache\") pod \"distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8\" (UID: \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:08:32.272351 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:32.272187 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-tmp-dir\") pod \"distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8\" (UID: \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:08:32.272351 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:32.272248 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmflk\" (UniqueName: \"kubernetes.io/projected/5e3275e5-f49f-4ca7-8d49-03178a77fee6-kube-api-access-mmflk\") pod \"distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8\" (UID: \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:08:32.272458 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:32.272444 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-home\") pod \"distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8\" (UID: \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:08:32.272712 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:32.272684 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-tmp-dir\") pod \"distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8\" (UID: \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:08:32.272980 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:32.272961 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-model-cache\") pod \"distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8\" (UID: \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:08:32.274573 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:32.274549 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-dshm\") pod \"distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8\" (UID: \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:08:32.275077 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:32.275054 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e3275e5-f49f-4ca7-8d49-03178a77fee6-tls-certs\") pod \"distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8\" (UID: \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:08:32.333120 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:32.333041 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmflk\" (UniqueName: \"kubernetes.io/projected/5e3275e5-f49f-4ca7-8d49-03178a77fee6-kube-api-access-mmflk\") pod \"distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8\" (UID: \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\") " pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:08:32.357599 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:32.357552 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:08:32.745906 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:32.745878 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8"] Nov 02 23:08:32.748180 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:08:32.748145 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e3275e5_f49f_4ca7_8d49_03178a77fee6.slice/crio-2ffeaf44ef293fd5bc05153ebaced2644a9ff36d4ff255154b9ea5242d2cb11f WatchSource:0}: Error finding container 2ffeaf44ef293fd5bc05153ebaced2644a9ff36d4ff255154b9ea5242d2cb11f: Status 404 returned error can't find the container with id 2ffeaf44ef293fd5bc05153ebaced2644a9ff36d4ff255154b9ea5242d2cb11f Nov 02 23:08:33.378595 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:33.378546 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" event={"ID":"5e3275e5-f49f-4ca7-8d49-03178a77fee6","Type":"ContainerStarted","Data":"04aad5caac6eeac5e11f5618e6c4a241b1d3afb1b178209739a9dda4e3a8e698"} Nov 02 23:08:33.379031 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:33.378604 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" event={"ID":"5e3275e5-f49f-4ca7-8d49-03178a77fee6","Type":"ContainerStarted","Data":"2ffeaf44ef293fd5bc05153ebaced2644a9ff36d4ff255154b9ea5242d2cb11f"} Nov 02 23:08:33.496401 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:33.496325 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" podStartSLOduration=1.496302395 podStartE2EDuration="1.496302395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:08:33.494599223 +0000 UTC m=+1188.847632598" watchObservedRunningTime="2026-11-02 23:08:33.496302395 +0000 UTC m=+1188.849335770" Nov 02 23:08:34.357704 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:34.357663 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:08:37.584925 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:08:37.584883 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:08:42.362632 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:42.362596 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:08:42.498778 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:42.498744 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:08:42.522471 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:08:42.522427 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:09:04.657968 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.657914 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx"] Nov 02 23:09:04.662187 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.662156 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:04.763655 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.763617 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx"] Nov 02 23:09:04.841169 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.841116 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:04.841169 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.841170 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49b8de5c-62d9-4246-a307-796410ed7bc0-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:04.841420 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.841192 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:04.841420 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.841315 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:04.841420 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.841354 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:04.841420 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.841382 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwp6h\" (UniqueName: \"kubernetes.io/projected/49b8de5c-62d9-4246-a307-796410ed7bc0-kube-api-access-lwp6h\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:04.841618 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.841475 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:04.841618 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.841578 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:04.841696 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.841624 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:04.942900 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.942811 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:04.942900 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.942856 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49b8de5c-62d9-4246-a307-796410ed7bc0-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:04.942900 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.942883 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:04.943173 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.942940 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:04.943173 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.942998 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:04.943173 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.943032 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwp6h\" (UniqueName: \"kubernetes.io/projected/49b8de5c-62d9-4246-a307-796410ed7bc0-kube-api-access-lwp6h\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:04.943173 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.943059 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:04.943393 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.943236 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:04.943393 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.943311 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:04.943393 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.943318 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:04.943393 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.943309 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:04.943393 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.943373 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:04.943908 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.943884 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:04.944028 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.943977 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49b8de5c-62d9-4246-a307-796410ed7bc0-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:04.946109 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.946086 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:04.946282 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:04.946256 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:05.004343 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:05.004315 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:05.004552 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:05.004531 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwp6h\" (UniqueName: \"kubernetes.io/projected/49b8de5c-62d9-4246-a307-796410ed7bc0-kube-api-access-lwp6h\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:05.278623 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:05.278514 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:05.718766 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:05.718741 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx"] Nov 02 23:09:05.721216 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:09:05.721184 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b8de5c_62d9_4246_a307_796410ed7bc0.slice/crio-3424f061151b2947655ad51edb147717d2d50cd7001d38eb7cce7354f4244c97 WatchSource:0}: Error finding container 3424f061151b2947655ad51edb147717d2d50cd7001d38eb7cce7354f4244c97: Status 404 returned error can't find the container with id 3424f061151b2947655ad51edb147717d2d50cd7001d38eb7cce7354f4244c97 Nov 02 23:09:05.724147 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:05.724095 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:09:05.724278 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:05.724185 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:09:05.724278 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:05.724232 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:09:06.604832 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:06.604797 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" event={"ID":"49b8de5c-62d9-4246-a307-796410ed7bc0","Type":"ContainerStarted","Data":"1ff52d90e3618a92c6fe8c9a85d49706b5d89b57f3623f9c52ccb6b4ece91c1d"} Nov 02 23:09:06.605046 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:06.604856 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" event={"ID":"49b8de5c-62d9-4246-a307-796410ed7bc0","Type":"ContainerStarted","Data":"3424f061151b2947655ad51edb147717d2d50cd7001d38eb7cce7354f4244c97"} Nov 02 23:09:06.778669 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:06.778614 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" podStartSLOduration=2.778596902 podStartE2EDuration="2.778596902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:09:06.77622632 +0000 UTC m=+1222.129259698" watchObservedRunningTime="2026-11-02 23:09:06.778596902 +0000 UTC m=+1222.131630276" Nov 02 23:09:07.279445 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:07.279404 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:07.281279 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:07.281242 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 23:09:07.281435 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:07.281313 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" podUID="49b8de5c-62d9-4246-a307-796410ed7bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 23:09:07.657699 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:09:07.657658 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:09:08.279147 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:08.279105 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 23:09:08.279598 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:08.279168 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" podUID="49b8de5c-62d9-4246-a307-796410ed7bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 23:09:09.279146 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:09.279097 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 23:09:09.279602 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:09.279163 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" podUID="49b8de5c-62d9-4246-a307-796410ed7bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 23:09:10.279628 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:10.279587 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 23:09:10.280066 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:10.279650 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" podUID="49b8de5c-62d9-4246-a307-796410ed7bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 23:09:11.279943 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:11.279905 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 23:09:11.280360 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:11.279968 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" podUID="49b8de5c-62d9-4246-a307-796410ed7bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 23:09:12.280047 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:12.280009 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 23:09:12.280650 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:12.280611 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" podUID="49b8de5c-62d9-4246-a307-796410ed7bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 23:09:13.601317 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:13.596559 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:13.601317 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:13.597288 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:13.601317 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:13.598169 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:09:37.585915 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:09:37.585875 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:09:40.309663 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:40.309607 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" podUID="496c6815-1b56-4082-9ffd-9ea51a762a58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:09:40.326656 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:40.326600 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" podUID="832b80ef-5c2c-4791-81c4-b92748d79552" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:09:40.403289 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:09:40.403237 5114 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" podUID="4d705cd8-8b43-4229-9df0-b0980e7728aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:03.641340 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.641305 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds"] Nov 02 23:10:03.644952 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.644932 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.666711 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.666672 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.666881 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.666807 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.666881 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.666864 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0879a805-4cb8-42a1-99f4-0e18d318335f-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.666987 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.666913 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.667070 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.667050 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.667129 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.667085 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.667209 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.667147 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.667209 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.667183 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.667286 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.667216 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9thqj\" (UniqueName: \"kubernetes.io/projected/0879a805-4cb8-42a1-99f4-0e18d318335f-kube-api-access-9thqj\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.765118 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.751053 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds"] Nov 02 23:10:03.768216 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.768189 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.768336 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.768258 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.768336 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.768278 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.768336 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.768298 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.768336 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.768328 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.768565 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.768371 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9thqj\" (UniqueName: \"kubernetes.io/projected/0879a805-4cb8-42a1-99f4-0e18d318335f-kube-api-access-9thqj\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.768565 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.768422 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.768565 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.768521 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.768565 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.768551 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0879a805-4cb8-42a1-99f4-0e18d318335f-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.768784 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.768673 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.768784 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.768714 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.768891 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.768794 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.769084 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.769062 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.769574 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.769552 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0879a805-4cb8-42a1-99f4-0e18d318335f-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.771369 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.771344 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.771536 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.771517 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.829144 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.829107 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.829355 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.829332 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9thqj\" (UniqueName: \"kubernetes.io/projected/0879a805-4cb8-42a1-99f4-0e18d318335f-kube-api-access-9thqj\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:03.960078 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:03.959986 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:04.399100 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:04.399069 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds"] Nov 02 23:10:04.400749 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:10:04.400711 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0879a805_4cb8_42a1_99f4_0e18d318335f.slice/crio-f28d8db9028d6fed2f67d62bc6614adc11231013e6ede1aeb78b4f4e01c6513c WatchSource:0}: Error finding container f28d8db9028d6fed2f67d62bc6614adc11231013e6ede1aeb78b4f4e01c6513c: Status 404 returned error can't find the container with id f28d8db9028d6fed2f67d62bc6614adc11231013e6ede1aeb78b4f4e01c6513c Nov 02 23:10:04.403133 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:04.403103 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:10:04.403198 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:04.403174 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:10:04.403237 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:04.403201 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:10:05.084222 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.084182 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" event={"ID":"0879a805-4cb8-42a1-99f4-0e18d318335f","Type":"ContainerStarted","Data":"af3b1c9186e95266421c013f47c408d17399e582e267950b726a1bf86d89a0b6"} Nov 02 23:10:05.084660 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.084251 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" event={"ID":"0879a805-4cb8-42a1-99f4-0e18d318335f","Type":"ContainerStarted","Data":"f28d8db9028d6fed2f67d62bc6614adc11231013e6ede1aeb78b4f4e01c6513c"} Nov 02 23:10:05.135223 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.135169 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b"] Nov 02 23:10:05.139526 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.139465 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.182277 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.182242 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b"] Nov 02 23:10:05.183793 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.183763 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.183932 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.183845 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.183932 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.183900 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml55g\" (UniqueName: \"kubernetes.io/projected/b45caa0e-af1c-45df-81af-f0ba37e27f16-kube-api-access-ml55g\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.184039 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.183930 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.184095 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.184037 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.184170 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.184147 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.184219 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.184205 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.184561 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.184537 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.184724 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.184696 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45caa0e-af1c-45df-81af-f0ba37e27f16-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.238119 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.238051 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" podStartSLOduration=2.238028942 podStartE2EDuration="2.238028942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:10:05.23675872 +0000 UTC m=+1280.589792095" watchObservedRunningTime="2026-11-02 23:10:05.238028942 +0000 UTC m=+1280.591062317" Nov 02 23:10:05.286329 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.286291 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.286561 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.286379 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.286561 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.286421 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.286561 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.286547 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.286755 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.286629 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45caa0e-af1c-45df-81af-f0ba37e27f16-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.286755 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.286694 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.286858 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.286758 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.286858 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.286808 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml55g\" (UniqueName: \"kubernetes.io/projected/b45caa0e-af1c-45df-81af-f0ba37e27f16-kube-api-access-ml55g\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.286858 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.286837 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.287019 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.286861 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.287019 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.286886 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.287195 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.287176 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.287260 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.287197 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.287444 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.287424 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45caa0e-af1c-45df-81af-f0ba37e27f16-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.289736 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.289710 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.289948 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.289930 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.349465 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.349391 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.349465 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.349398 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml55g\" (UniqueName: \"kubernetes.io/projected/b45caa0e-af1c-45df-81af-f0ba37e27f16-kube-api-access-ml55g\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.454063 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.454024 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:05.903381 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.903349 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b"] Nov 02 23:10:05.905237 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:10:05.905204 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45caa0e_af1c_45df_81af_f0ba37e27f16.slice/crio-ecf1d7cfd94a4cd673ca4671daa677752fa7f7598edffff9a0b502c755cc4927 WatchSource:0}: Error finding container ecf1d7cfd94a4cd673ca4671daa677752fa7f7598edffff9a0b502c755cc4927: Status 404 returned error can't find the container with id ecf1d7cfd94a4cd673ca4671daa677752fa7f7598edffff9a0b502c755cc4927 Nov 02 23:10:05.931789 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.924006 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:10:05.931789 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.924105 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:10:05.931789 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.924150 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:10:05.961182 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.961150 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:05.962904 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.962864 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 23:10:05.963050 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:05.962933 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" podUID="0879a805-4cb8-42a1-99f4-0e18d318335f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 23:10:06.098758 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:06.098639 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" event={"ID":"b45caa0e-af1c-45df-81af-f0ba37e27f16","Type":"ContainerStarted","Data":"55736e5fcb0c962a4a764c4cf6a3ca9d995aa5e22932b3f1a614f15aba13c870"} Nov 02 23:10:06.098758 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:06.098702 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" event={"ID":"b45caa0e-af1c-45df-81af-f0ba37e27f16","Type":"ContainerStarted","Data":"ecf1d7cfd94a4cd673ca4671daa677752fa7f7598edffff9a0b502c755cc4927"} Nov 02 23:10:06.226577 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:06.222062 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" podStartSLOduration=1.222037549 podStartE2EDuration="1.222037549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:10:06.219982149 +0000 UTC m=+1281.573015526" watchObservedRunningTime="2026-11-02 23:10:06.222037549 +0000 UTC m=+1281.575070926" Nov 02 23:10:06.454715 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:06.454669 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:06.971508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:06.970887 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 23:10:06.971508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:06.970984 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" podUID="0879a805-4cb8-42a1-99f4-0e18d318335f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 23:10:07.466509 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:07.462230 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 23:10:07.466509 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:07.462317 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" podUID="b45caa0e-af1c-45df-81af-f0ba37e27f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 23:10:07.799076 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:10:07.799035 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:10:07.878639 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:07.878585 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" podUID="4d705cd8-8b43-4229-9df0-b0980e7728aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 23:10:07.972059 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:07.972013 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 23:10:07.972363 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:07.972331 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" podUID="0879a805-4cb8-42a1-99f4-0e18d318335f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 23:10:08.177506 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:08.174285 5114 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" podUID="5efc1b0f-5388-4059-b8da-5ae6c7a48501" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:08.177506 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:08.174684 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" podUID="f850c9fa-e500-4772-8454-a5172e446770" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 23:10:08.313853 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:08.312584 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" podUID="496c6815-1b56-4082-9ffd-9ea51a762a58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:08.327737 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:08.327685 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" podUID="832b80ef-5c2c-4791-81c4-b92748d79552" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:08.457271 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:08.457162 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" podUID="592a8e63-9584-4890-b38e-900b0482f735" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:08.458596 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:08.458565 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 23:10:08.458851 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:08.458827 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" podUID="b45caa0e-af1c-45df-81af-f0ba37e27f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 23:10:08.495638 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:08.495584 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" podUID="fb16fc7d-3def-41dc-91a7-1103ccaf4c35" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 02 23:10:08.508646 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:08.508589 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" podUID="5e3275e5-f49f-4ca7-8d49-03178a77fee6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:08.567616 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:08.567569 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" podUID="bcd6362b-2995-4ebc-972d-dfcee6e41357" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Nov 02 23:10:08.623691 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:08.623644 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" podUID="e9da41e7-2e4b-4912-9267-3f732f714155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:08.667711 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:08.667649 5114 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" podUID="fb16fc7d-3def-41dc-91a7-1103ccaf4c35" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:08.714783 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:08.714649 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" podUID="72301e09-ed03-4bc9-8264-028de7930a48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:08.804730 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:08.804668 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" podUID="9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 23:10:08.807031 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:08.805908 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" podUID="5efc1b0f-5388-4059-b8da-5ae6c7a48501" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:08.882675 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:08.882616 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" podUID="4d705cd8-8b43-4229-9df0-b0980e7728aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:08.883526 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:08.883471 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:10:08.961104 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:08.961055 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 23:10:08.961420 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:08.961374 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" podUID="0879a805-4cb8-42a1-99f4-0e18d318335f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 23:10:09.216758 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.216710 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" podUID="f850c9fa-e500-4772-8454-a5172e446770" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 23:10:09.236705 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.236664 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:10:09.282511 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.252293 5114 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" podUID="592a8e63-9584-4890-b38e-900b0482f735" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:09.333518 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.323636 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" podUID="496c6815-1b56-4082-9ffd-9ea51a762a58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:09.333518 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.324122 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:10:09.388513 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.333912 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" podUID="832b80ef-5c2c-4791-81c4-b92748d79552" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded" Nov 02 23:10:09.388513 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.334355 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 23:10:09.476717 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.476614 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" podUID="592a8e63-9584-4890-b38e-900b0482f735" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:09.502813 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.502759 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" podUID="fb16fc7d-3def-41dc-91a7-1103ccaf4c35" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:09.503914 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.503883 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 23:10:09.504118 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.504093 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" podUID="b45caa0e-af1c-45df-81af-f0ba37e27f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 23:10:09.580511 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.577721 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:10:09.580511 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.577826 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:10:09.634292 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.634248 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" podUID="e9da41e7-2e4b-4912-9267-3f732f714155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Nov 02 23:10:09.667668 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.635353 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" podUID="5e3275e5-f49f-4ca7-8d49-03178a77fee6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 23:10:09.714135 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.655716 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" podUID="bcd6362b-2995-4ebc-972d-dfcee6e41357" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:09.714529 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.668766 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:10:09.714783 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.714766 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:10:09.714977 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.714946 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:10:09.748851 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.748760 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" podUID="72301e09-ed03-4bc9-8264-028de7930a48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:09.749250 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.749227 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:10:09.895525 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.892178 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" podUID="4d705cd8-8b43-4229-9df0-b0980e7728aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:09.994544 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.970018 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" podUID="5efc1b0f-5388-4059-b8da-5ae6c7a48501" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:09.994544 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.970532 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:10:09.997129 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.997082 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 23:10:09.997405 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:09.997357 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" podUID="0879a805-4cb8-42a1-99f4-0e18d318335f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 23:10:10.111525 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:10.108222 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 02 23:10:10.226334 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:10.226288 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" podUID="f850c9fa-e500-4772-8454-a5172e446770" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 23:10:10.253173 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:10.253137 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 02 23:10:10.362658 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:10.327592 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" podUID="496c6815-1b56-4082-9ffd-9ea51a762a58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:10.373508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:10.359122 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 02 23:10:10.413614 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:10.359672 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" podUID="832b80ef-5c2c-4791-81c4-b92748d79552" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded" Nov 02 23:10:10.458734 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:10.458700 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 23:10:10.472644 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:10.462736 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" podUID="b45caa0e-af1c-45df-81af-f0ba37e27f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 23:10:10.505513 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:10.485363 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" podUID="592a8e63-9584-4890-b38e-900b0482f735" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 02 23:10:10.791524 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:10.788703 5114 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" podUID="4d705cd8-8b43-4229-9df0-b0980e7728aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded" Nov 02 23:10:10.791524 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:10.789047 5114 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" podUID="72301e09-ed03-4bc9-8264-028de7930a48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:10.791524 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:10.789173 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" podUID="e9da41e7-2e4b-4912-9267-3f732f714155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Nov 02 23:10:10.791524 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:10.790304 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" podUID="72301e09-ed03-4bc9-8264-028de7930a48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:10.837656 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:10.837605 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" podUID="9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:10.901161 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:10.901121 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" podUID="4d705cd8-8b43-4229-9df0-b0980e7728aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:11.005771 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:11.005716 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" podUID="5efc1b0f-5388-4059-b8da-5ae6c7a48501" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 23:10:11.184573 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:11.128651 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 23:10:11.194511 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:11.184827 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" podUID="0879a805-4cb8-42a1-99f4-0e18d318335f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 23:10:11.273016 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:11.272961 5114 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" podUID="f850c9fa-e500-4772-8454-a5172e446770" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Nov 02 23:10:11.275275 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:11.275227 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" podUID="f850c9fa-e500-4772-8454-a5172e446770" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:11.287503 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:11.282850 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" podUID="bcd6362b-2995-4ebc-972d-dfcee6e41357" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:11.304501 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:11.296510 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" Nov 02 23:10:11.304501 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:11.296578 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 02 23:10:11.304501 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:11.296614 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 02 23:10:11.304501 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:11.298242 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 02 23:10:11.304501 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:11.300801 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:10:11.304501 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:11.300851 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 02 23:10:11.363444 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:11.362692 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" podUID="fb16fc7d-3def-41dc-91a7-1103ccaf4c35" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 02 23:10:11.379087 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:11.379056 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 02 23:10:11.383924 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:11.383887 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 02 23:10:11.458460 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:11.458334 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 23:10:11.458796 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:11.458745 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" podUID="b45caa0e-af1c-45df-81af-f0ba37e27f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 23:10:11.964939 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:11.964899 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 23:10:11.965622 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:11.965585 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" podUID="0879a805-4cb8-42a1-99f4-0e18d318335f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 23:10:12.462584 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:12.462533 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 23:10:12.463531 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:12.463151 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" podUID="b45caa0e-af1c-45df-81af-f0ba37e27f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 23:10:12.961073 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:12.961033 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 23:10:12.962651 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:12.961951 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" podUID="0879a805-4cb8-42a1-99f4-0e18d318335f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 23:10:13.455237 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:13.455200 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 23:10:13.455809 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:13.455263 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" podUID="b45caa0e-af1c-45df-81af-f0ba37e27f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 23:10:14.455035 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:14.454985 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 23:10:14.455235 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:14.455062 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" podUID="b45caa0e-af1c-45df-81af-f0ba37e27f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 23:10:14.897527 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:14.897466 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:14.898049 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:14.897962 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:14.898873 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:14.898849 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:10:15.454456 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:15.454418 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 23:10:15.454678 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:15.454544 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" podUID="b45caa0e-af1c-45df-81af-f0ba37e27f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 23:10:16.474605 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:16.474572 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:16.475431 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:16.475403 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:16.476541 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:16.476521 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:10:32.847516 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:32.842803 5114 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" podUID="bcd6362b-2995-4ebc-972d-dfcee6e41357" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:32.900222 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:32.849609 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" podUID="9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:32.912046 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:32.911590 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" podUID="4d705cd8-8b43-4229-9df0-b0980e7728aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:32.986630 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:32.982173 5114 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" podUID="e9da41e7-2e4b-4912-9267-3f732f714155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": context deadline exceeded" Nov 02 23:10:33.112744 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:33.112599 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" podUID="5e3275e5-f49f-4ca7-8d49-03178a77fee6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:33.113456 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:33.113420 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" podUID="5efc1b0f-5388-4059-b8da-5ae6c7a48501" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:33.216633 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:33.216580 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" podUID="72301e09-ed03-4bc9-8264-028de7930a48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:33.258238 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:33.257328 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" podUID="bcd6362b-2995-4ebc-972d-dfcee6e41357" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:33.283732 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:33.283672 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" podUID="e9da41e7-2e4b-4912-9267-3f732f714155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:33.359420 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:33.357976 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" podUID="f850c9fa-e500-4772-8454-a5172e446770" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:37.584430 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:10:37.584390 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:10:50.249340 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:50.249304 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh"] Nov 02 23:10:50.253400 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:50.253373 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 02 23:10:50.318884 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:50.318836 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-08c2c6\"/\"distinct-2-w2-08c2c6-kserve-self-signed-certs\"" Nov 02 23:10:50.321687 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:50.321641 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-home\") pod \"distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh\" (UID: \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 02 23:10:50.321897 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:50.321753 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-dshm\") pod \"distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh\" (UID: \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 02 23:10:50.321897 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:50.321800 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h5mx\" (UniqueName: \"kubernetes.io/projected/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-kube-api-access-6h5mx\") pod \"distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh\" (UID: \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 02 23:10:50.321897 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:50.321863 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-model-cache\") pod \"distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh\" (UID: \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 02 23:10:50.322077 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:50.322025 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-tmp-dir\") pod \"distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh\" (UID: \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 02 23:10:50.322122 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:50.322084 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-tls-certs\") pod \"distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh\" (UID: \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 02 23:10:50.323218 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:50.323191 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh"] Nov 02 23:10:50.423649 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:50.423606 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-home\") pod \"distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh\" (UID: \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 02 23:10:50.423649 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:50.423655 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-dshm\") pod \"distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh\" (UID: \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 02 23:10:50.423924 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:50.423698 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h5mx\" (UniqueName: \"kubernetes.io/projected/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-kube-api-access-6h5mx\") pod \"distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh\" (UID: \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 02 23:10:50.423924 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:50.423740 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-model-cache\") pod \"distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh\" (UID: \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 02 23:10:50.423924 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:50.423768 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-tmp-dir\") pod \"distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh\" (UID: \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 02 23:10:50.423924 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:50.423807 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-tls-certs\") pod \"distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh\" (UID: \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 02 23:10:50.424133 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:50.424081 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-home\") pod \"distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh\" (UID: \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 02 23:10:50.424236 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:50.424213 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-tmp-dir\") pod \"distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh\" (UID: \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 02 23:10:50.424510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:50.424467 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-model-cache\") pod \"distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh\" (UID: \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 02 23:10:50.426357 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:50.426330 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-dshm\") pod \"distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh\" (UID: \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 02 23:10:50.426629 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:50.426607 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-tls-certs\") pod \"distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh\" (UID: \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 02 23:10:50.484994 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:50.484942 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h5mx\" (UniqueName: \"kubernetes.io/projected/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-kube-api-access-6h5mx\") pod \"distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh\" (UID: \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\") " pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 02 23:10:50.568735 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:50.568682 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 02 23:10:50.694988 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:50.694884 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c"] Nov 02 23:10:50.700836 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:50.699823 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" podUID="45a38607-6428-49c6-ab92-eb788f47a236" containerName="istio-proxy" containerID="cri-o://d07ec33e90efc5e6926b5f4687266b352ce875d0da2c9d0cb59f326966e6b4e6" gracePeriod=30 Nov 02 23:10:51.027452 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.027425 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh"] Nov 02 23:10:51.030248 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:10:51.030217 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e5fe23d_d96a_4099_a9bc_7844e2da3eb1.slice/crio-7a83e18ed65c7ff6b1ca4fae6daae381a61431a46de119d8a022685056a19773 WatchSource:0}: Error finding container 7a83e18ed65c7ff6b1ca4fae6daae381a61431a46de119d8a022685056a19773: Status 404 returned error can't find the container with id 7a83e18ed65c7ff6b1ca4fae6daae381a61431a46de119d8a022685056a19773 Nov 02 23:10:51.439110 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.439067 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" event={"ID":"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1","Type":"ContainerStarted","Data":"d54e5edf32783c9861e1ff0237cc25adc511c6f78fa62c56f86438a4383d7f66"} Nov 02 23:10:51.439671 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.439129 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" event={"ID":"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1","Type":"ContainerStarted","Data":"7a83e18ed65c7ff6b1ca4fae6daae381a61431a46de119d8a022685056a19773"} Nov 02 23:10:51.557628 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.557572 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" podStartSLOduration=1.55755753 podStartE2EDuration="1.55755753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:10:51.556133128 +0000 UTC m=+1326.909166503" watchObservedRunningTime="2026-11-02 23:10:51.55755753 +0000 UTC m=+1326.910590904" Nov 02 23:10:51.568985 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.568932 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 02 23:10:51.617779 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.617746 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm"] Nov 02 23:10:51.621524 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.621500 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.721115 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.721020 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm"] Nov 02 23:10:51.741545 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.741500 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-credential-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.741757 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.741599 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-workload-certs\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.741757 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.741724 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-token\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.741886 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.741784 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-podinfo\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.741886 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.741808 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade938ea-2fe8-49c9-908f-78bbf9b02606-istiod-ca-cert\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.741886 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.741852 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-envoy\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.742021 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.742000 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-workload-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.742072 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.742031 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49fm8\" (UniqueName: \"kubernetes.io/projected/ade938ea-2fe8-49c9-908f-78bbf9b02606-kube-api-access-49fm8\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.742072 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.742062 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-data\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.843546 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.843477 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-token\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.843767 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.843587 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-podinfo\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.843767 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.843615 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade938ea-2fe8-49c9-908f-78bbf9b02606-istiod-ca-cert\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.843767 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.843662 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-envoy\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.843767 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.843724 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-workload-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.843767 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.843749 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49fm8\" (UniqueName: \"kubernetes.io/projected/ade938ea-2fe8-49c9-908f-78bbf9b02606-kube-api-access-49fm8\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.844093 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.843791 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-data\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.844093 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.843830 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-credential-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.844093 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.843875 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-workload-certs\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.844321 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.844288 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-workload-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.844391 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.844356 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-data\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.844391 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.844370 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-workload-certs\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.844700 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.844435 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade938ea-2fe8-49c9-908f-78bbf9b02606-istiod-ca-cert\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.844700 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.844659 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-credential-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.846457 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.846424 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-envoy\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.846655 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.846589 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-podinfo\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.904739 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.904709 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-token\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.904929 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.904911 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49fm8\" (UniqueName: \"kubernetes.io/projected/ade938ea-2fe8-49c9-908f-78bbf9b02606-kube-api-access-49fm8\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:51.937068 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:51.937032 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:52.371040 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:52.370968 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm"] Nov 02 23:10:52.373642 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:10:52.373608 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade938ea_2fe8_49c9_908f_78bbf9b02606.slice/crio-3c8cd7a11a51e8ffe64e6f7d2c6a6ff762128b30c4b483c15aaf7587dbf62501 WatchSource:0}: Error finding container 3c8cd7a11a51e8ffe64e6f7d2c6a6ff762128b30c4b483c15aaf7587dbf62501: Status 404 returned error can't find the container with id 3c8cd7a11a51e8ffe64e6f7d2c6a6ff762128b30c4b483c15aaf7587dbf62501 Nov 02 23:10:52.376332 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:52.376298 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:10:52.376426 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:52.376369 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:10:52.376426 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:52.376400 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:10:52.447557 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:52.447513 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" event={"ID":"ade938ea-2fe8-49c9-908f-78bbf9b02606","Type":"ContainerStarted","Data":"ce45ea75367581746591b8134e028b1b2a1c979f8b83c3a331cc98f6cc0a0058"} Nov 02 23:10:52.448004 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:52.447576 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" event={"ID":"ade938ea-2fe8-49c9-908f-78bbf9b02606","Type":"ContainerStarted","Data":"3c8cd7a11a51e8ffe64e6f7d2c6a6ff762128b30c4b483c15aaf7587dbf62501"} Nov 02 23:10:52.568230 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:52.568162 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" podStartSLOduration=1.568141873 podStartE2EDuration="1.568141873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:10:52.566280411 +0000 UTC m=+1327.919313788" watchObservedRunningTime="2026-11-02 23:10:52.568141873 +0000 UTC m=+1327.921175354" Nov 02 23:10:52.938207 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:52.938152 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:10:53.947509 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:53.946774 5114 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 23:10:53.947509 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:53.946853 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" podUID="ade938ea-2fe8-49c9-908f-78bbf9b02606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 23:10:54.940514 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:54.940029 5114 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 23:10:54.940514 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:54.940090 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" podUID="ade938ea-2fe8-49c9-908f-78bbf9b02606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 23:10:55.043508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:55.043115 5114 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" podUID="496c6815-1b56-4082-9ffd-9ea51a762a58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:55.116510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:55.115451 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" podUID="5e3275e5-f49f-4ca7-8d49-03178a77fee6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:55.258572 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:55.256823 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" podUID="bcd6362b-2995-4ebc-972d-dfcee6e41357" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:55.328506 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:55.324636 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" podUID="e9da41e7-2e4b-4912-9267-3f732f714155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:55.328506 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:55.325566 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" podUID="f850c9fa-e500-4772-8454-a5172e446770" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:55.328506 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:55.326201 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" podUID="496c6815-1b56-4082-9ffd-9ea51a762a58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 02 23:10:55.380443 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:55.380393 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" podUID="fb16fc7d-3def-41dc-91a7-1103ccaf4c35" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:55.588837 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:55.588574 5114 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4" podUID="832b80ef-5c2c-4791-81c4-b92748d79552" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:55.941435 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:55.940955 5114 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 23:10:55.941435 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:55.941044 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" podUID="ade938ea-2fe8-49c9-908f-78bbf9b02606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 23:10:56.483076 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.482562 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" podUID="f850c9fa-e500-4772-8454-a5172e446770" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:56.484904 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.484096 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:10:56.596663 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.596402 5114 generic.go:356] "Generic (PLEG): container finished" podID="45a38607-6428-49c6-ab92-eb788f47a236" containerID="d07ec33e90efc5e6926b5f4687266b352ce875d0da2c9d0cb59f326966e6b4e6" exitCode=0 Nov 02 23:10:56.596871 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.596722 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" event={"ID":"45a38607-6428-49c6-ab92-eb788f47a236","Type":"ContainerDied","Data":"d07ec33e90efc5e6926b5f4687266b352ce875d0da2c9d0cb59f326966e6b4e6"} Nov 02 23:10:56.596871 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.596808 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" event={"ID":"45a38607-6428-49c6-ab92-eb788f47a236","Type":"ContainerDied","Data":"1cece0912f2bba46826120e8ec59410d4c57d2bfb2c638bd63be7288235af570"} Nov 02 23:10:56.596871 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.596863 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cece0912f2bba46826120e8ec59410d4c57d2bfb2c638bd63be7288235af570" Nov 02 23:10:56.597237 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.597216 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 02 23:10:56.610505 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.607707 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:10:56.791863 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.791829 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a38607-6428-49c6-ab92-eb788f47a236-kube-api-access-fxkgb\" (UniqueName: \"kubernetes.io/projected/45a38607-6428-49c6-ab92-eb788f47a236-kube-api-access-fxkgb\") pod \"45a38607-6428-49c6-ab92-eb788f47a236\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " Nov 02 23:10:56.792065 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.791888 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-credential-socket\") pod \"45a38607-6428-49c6-ab92-eb788f47a236\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " Nov 02 23:10:56.792065 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.791966 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-istio-envoy\") pod \"45a38607-6428-49c6-ab92-eb788f47a236\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " Nov 02 23:10:56.792065 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.792043 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45a38607-6428-49c6-ab92-eb788f47a236-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a38607-6428-49c6-ab92-eb788f47a236-istio-podinfo\") pod \"45a38607-6428-49c6-ab92-eb788f47a236\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " Nov 02 23:10:56.792196 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.792116 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a38607-6428-49c6-ab92-eb788f47a236-istio-token\" (UniqueName: \"kubernetes.io/projected/45a38607-6428-49c6-ab92-eb788f47a236-istio-token\") pod \"45a38607-6428-49c6-ab92-eb788f47a236\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " Nov 02 23:10:56.792196 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.792146 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-workload-socket\") pod \"45a38607-6428-49c6-ab92-eb788f47a236\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " Nov 02 23:10:56.792298 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.792255 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-workload-certs\") pod \"45a38607-6428-49c6-ab92-eb788f47a236\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " Nov 02 23:10:56.792298 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.792288 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-istio-data\") pod \"45a38607-6428-49c6-ab92-eb788f47a236\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " Nov 02 23:10:56.792394 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.792315 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45a38607-6428-49c6-ab92-eb788f47a236-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a38607-6428-49c6-ab92-eb788f47a236-istiod-ca-cert\") pod \"45a38607-6428-49c6-ab92-eb788f47a236\" (UID: \"45a38607-6428-49c6-ab92-eb788f47a236\") " Nov 02 23:10:56.792689 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.792660 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-credential-socket" pod "45a38607-6428-49c6-ab92-eb788f47a236" (UID: "45a38607-6428-49c6-ab92-eb788f47a236"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:10:56.792961 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.792940 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:56.794825 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.794795 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a38607-6428-49c6-ab92-eb788f47a236-kube-api-access-fxkgb" pod "45a38607-6428-49c6-ab92-eb788f47a236" (UID: "45a38607-6428-49c6-ab92-eb788f47a236"). InnerVolumeSpecName "kube-api-access-fxkgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:10:56.795445 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.795423 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45a38607-6428-49c6-ab92-eb788f47a236-istio-podinfo" pod "45a38607-6428-49c6-ab92-eb788f47a236" (UID: "45a38607-6428-49c6-ab92-eb788f47a236"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:10:56.795823 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.795803 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-istio-data" pod "45a38607-6428-49c6-ab92-eb788f47a236" (UID: "45a38607-6428-49c6-ab92-eb788f47a236"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:10:56.796109 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.796091 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-workload-socket" pod "45a38607-6428-49c6-ab92-eb788f47a236" (UID: "45a38607-6428-49c6-ab92-eb788f47a236"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:10:56.796367 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.796349 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-workload-certs" pod "45a38607-6428-49c6-ab92-eb788f47a236" (UID: "45a38607-6428-49c6-ab92-eb788f47a236"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:10:56.796731 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.796703 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a38607-6428-49c6-ab92-eb788f47a236-istiod-ca-cert" pod "45a38607-6428-49c6-ab92-eb788f47a236" (UID: "45a38607-6428-49c6-ab92-eb788f47a236"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:10:56.797020 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.797000 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-istio-envoy" pod "45a38607-6428-49c6-ab92-eb788f47a236" (UID: "45a38607-6428-49c6-ab92-eb788f47a236"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:10:56.803977 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.803939 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a38607-6428-49c6-ab92-eb788f47a236-istio-token" pod "45a38607-6428-49c6-ab92-eb788f47a236" (UID: "45a38607-6428-49c6-ab92-eb788f47a236"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:10:56.893994 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.893960 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:56.893994 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.893990 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a38607-6428-49c6-ab92-eb788f47a236-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:56.893994 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.894000 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45a38607-6428-49c6-ab92-eb788f47a236-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:56.894254 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.894009 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:56.894254 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.894018 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:56.894254 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.894025 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a38607-6428-49c6-ab92-eb788f47a236-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:56.894254 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.894037 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a38607-6428-49c6-ab92-eb788f47a236-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:56.894254 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.894047 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxkgb\" (UniqueName: \"kubernetes.io/projected/45a38607-6428-49c6-ab92-eb788f47a236-kube-api-access-fxkgb\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:56.940942 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.940908 5114 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 23:10:56.941132 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:56.940966 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" podUID="ade938ea-2fe8-49c9-908f-78bbf9b02606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 23:10:57.602929 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:57.602895 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c" Nov 02 23:10:57.776349 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:57.776312 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c"] Nov 02 23:10:57.782197 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:57.782162 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-ppw7c"] Nov 02 23:10:57.937766 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:57.937676 5114 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 23:10:57.937766 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:57.937733 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" podUID="ade938ea-2fe8-49c9-908f-78bbf9b02606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 23:10:58.801018 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:58.800955 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45a38607-6428-49c6-ab92-eb788f47a236" path="/var/lib/kubelet/pods/45a38607-6428-49c6-ab92-eb788f47a236/volumes" Nov 02 23:10:58.938047 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:58.938012 5114 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 23:10:58.938332 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:58.938305 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" podUID="ade938ea-2fe8-49c9-908f-78bbf9b02606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 23:10:59.937994 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:59.937937 5114 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 23:10:59.938613 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:10:59.937999 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" podUID="ade938ea-2fe8-49c9-908f-78bbf9b02606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 23:11:00.569930 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:11:00.569884 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 02 23:11:00.586390 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:11:00.586353 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 02 23:11:00.637730 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:11:00.637695 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 02 23:11:00.942905 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:11:00.942818 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:11:00.943401 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:11:00.943226 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:11:00.943953 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:11:00.943934 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:11:07.584705 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:11:07.584665 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:11:37.585039 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:11:37.584992 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:11:44.966510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:11:44.959639 5114 scope.go:122] "RemoveContainer" containerID="d07ec33e90efc5e6926b5f4687266b352ce875d0da2c9d0cb59f326966e6b4e6" Nov 02 23:12:07.584752 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:12:07.584711 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:12:13.593404 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.593353 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9"] Nov 02 23:12:13.594157 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.594134 5114 cpu_manager.go:414] "containerMap: removing container" podUID="45a38607-6428-49c6-ab92-eb788f47a236" containerName="istio-proxy" Nov 02 23:12:13.594157 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.594157 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a38607-6428-49c6-ab92-eb788f47a236" containerName="istio-proxy" Nov 02 23:12:13.594395 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.594350 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a38607-6428-49c6-ab92-eb788f47a236" containerName="istio-proxy" Nov 02 23:12:13.597749 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.597721 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:12:13.699272 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.699232 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a7a67875\"/\"default-dockercfg-9fxtm\"" Nov 02 23:12:13.699272 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.699232 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a7a67875\"/\"openshift-service-ca.crt\"" Nov 02 23:12:13.699272 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.699243 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a7a67875\"/\"kube-root-ca.crt\"" Nov 02 23:12:13.699558 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.699419 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a7a67875\"/\"e2e-lifecycle-model-a7a67875-kserve-self-signed-certs\"" Nov 02 23:12:13.703552 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.703523 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9"] Nov 02 23:12:13.741572 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.741539 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-model-cache\") pod \"e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9\" (UID: \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\") " pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:12:13.741781 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.741625 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-home\") pod \"e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9\" (UID: \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\") " pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:12:13.741845 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.741786 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-tls-certs\") pod \"e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9\" (UID: \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\") " pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:12:13.741995 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.741973 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-dshm\") pod \"e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9\" (UID: \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\") " pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:12:13.742065 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.742047 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-tmp-dir\") pod \"e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9\" (UID: \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\") " pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:12:13.742134 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.742110 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvst6\" (UniqueName: \"kubernetes.io/projected/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-kube-api-access-pvst6\") pod \"e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9\" (UID: \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\") " pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:12:13.843973 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.843860 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-tmp-dir\") pod \"e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9\" (UID: \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\") " pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:12:13.843973 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.843919 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvst6\" (UniqueName: \"kubernetes.io/projected/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-kube-api-access-pvst6\") pod \"e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9\" (UID: \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\") " pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:12:13.844261 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.844057 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-model-cache\") pod \"e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9\" (UID: \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\") " pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:12:13.844261 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.844138 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-home\") pod \"e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9\" (UID: \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\") " pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:12:13.844261 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.844200 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-tls-certs\") pod \"e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9\" (UID: \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\") " pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:12:13.844420 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.844274 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-dshm\") pod \"e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9\" (UID: \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\") " pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:12:13.844420 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.844305 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-tmp-dir\") pod \"e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9\" (UID: \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\") " pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:12:13.844916 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.844889 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-home\") pod \"e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9\" (UID: \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\") " pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:12:13.845069 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.844929 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-model-cache\") pod \"e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9\" (UID: \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\") " pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:12:13.847176 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.847156 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-dshm\") pod \"e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9\" (UID: \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\") " pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:12:13.847555 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.847537 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-tls-certs\") pod \"e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9\" (UID: \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\") " pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:12:13.905711 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.905675 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvst6\" (UniqueName: \"kubernetes.io/projected/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-kube-api-access-pvst6\") pod \"e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9\" (UID: \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\") " pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:12:13.910582 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:13.910544 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:12:14.296868 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:14.296836 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9"] Nov 02 23:12:14.298461 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:12:14.298431 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8f5d813_1cf7_47f0_b45e_230a313e7a1f.slice/crio-e6bf2877bc0f76c6c532c595ead6edd4ee7fc0049c2a2d9bed2ec2c383e06e3c WatchSource:0}: Error finding container e6bf2877bc0f76c6c532c595ead6edd4ee7fc0049c2a2d9bed2ec2c383e06e3c: Status 404 returned error can't find the container with id e6bf2877bc0f76c6c532c595ead6edd4ee7fc0049c2a2d9bed2ec2c383e06e3c Nov 02 23:12:15.189491 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:15.189450 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" event={"ID":"d8f5d813-1cf7-47f0-b45e-230a313e7a1f","Type":"ContainerStarted","Data":"0e7ef3e88564cc4b1765203fe4d82c0ba4109cdfa449f470532717a9806ed5fc"} Nov 02 23:12:15.189915 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:15.189521 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" event={"ID":"d8f5d813-1cf7-47f0-b45e-230a313e7a1f","Type":"ContainerStarted","Data":"e6bf2877bc0f76c6c532c595ead6edd4ee7fc0049c2a2d9bed2ec2c383e06e3c"} Nov 02 23:12:15.911606 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:15.911562 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:12:23.910963 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:23.910924 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:12:23.923763 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:23.923732 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:12:24.043972 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:24.043912 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" podStartSLOduration=11.04389485 podStartE2EDuration="11.04389485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:15.306716945 +0000 UTC m=+1410.659750323" watchObservedRunningTime="2026-11-02 23:12:24.04389485 +0000 UTC m=+1419.396928228" Nov 02 23:12:24.249954 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:24.249862 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:12:37.584988 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:12:37.584936 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:12:51.426168 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.426129 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl"] Nov 02 23:12:51.430646 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.430621 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:12:51.482733 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.482698 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 23:12:51.482938 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.482807 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rdqkr\"" Nov 02 23:12:51.482938 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.482746 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 23:12:51.482938 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.482747 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 23:12:51.536040 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.536001 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4da1367-063f-4955-9313-26c9e7b97729-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zr4hl\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:12:51.536236 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.536106 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d4vm\" (UniqueName: \"kubernetes.io/projected/a4da1367-063f-4955-9313-26c9e7b97729-kube-api-access-5d4vm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zr4hl\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:12:51.536236 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.536222 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a4da1367-063f-4955-9313-26c9e7b97729-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zr4hl\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:12:51.536354 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.536275 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a4da1367-063f-4955-9313-26c9e7b97729-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zr4hl\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:12:51.536408 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.536354 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4da1367-063f-4955-9313-26c9e7b97729-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zr4hl\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:12:51.536518 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.536494 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a4da1367-063f-4955-9313-26c9e7b97729-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zr4hl\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:12:51.536669 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.536555 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a4da1367-063f-4955-9313-26c9e7b97729-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zr4hl\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:12:51.536851 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.536834 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl"] Nov 02 23:12:51.638413 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.638369 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a4da1367-063f-4955-9313-26c9e7b97729-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zr4hl\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:12:51.638413 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.638421 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a4da1367-063f-4955-9313-26c9e7b97729-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zr4hl\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:12:51.638701 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.638541 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4da1367-063f-4955-9313-26c9e7b97729-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zr4hl\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:12:51.638701 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.638566 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d4vm\" (UniqueName: \"kubernetes.io/projected/a4da1367-063f-4955-9313-26c9e7b97729-kube-api-access-5d4vm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zr4hl\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:12:51.638701 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.638644 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a4da1367-063f-4955-9313-26c9e7b97729-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zr4hl\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:12:51.638701 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.638684 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a4da1367-063f-4955-9313-26c9e7b97729-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zr4hl\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:12:51.639101 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.638713 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4da1367-063f-4955-9313-26c9e7b97729-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zr4hl\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:12:51.639865 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.639832 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a4da1367-063f-4955-9313-26c9e7b97729-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zr4hl\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:12:51.641412 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.641381 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4da1367-063f-4955-9313-26c9e7b97729-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zr4hl\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:12:51.641731 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.641705 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a4da1367-063f-4955-9313-26c9e7b97729-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zr4hl\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:12:51.641830 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.641754 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a4da1367-063f-4955-9313-26c9e7b97729-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zr4hl\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:12:51.641830 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.641818 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a4da1367-063f-4955-9313-26c9e7b97729-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zr4hl\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:12:51.699866 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.699786 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4da1367-063f-4955-9313-26c9e7b97729-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zr4hl\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:12:51.700011 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.699921 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d4vm\" (UniqueName: \"kubernetes.io/projected/a4da1367-063f-4955-9313-26c9e7b97729-kube-api-access-5d4vm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zr4hl\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:12:51.743941 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:51.743906 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:12:52.175576 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:52.175547 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl"] Nov 02 23:12:52.178809 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:12:52.178766 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4da1367_063f_4955_9313_26c9e7b97729.slice/crio-41b7a4de3e1554dcbf3f737ac6a27223be383d86091ac421e08d194d1b061f80 WatchSource:0}: Error finding container 41b7a4de3e1554dcbf3f737ac6a27223be383d86091ac421e08d194d1b061f80: Status 404 returned error can't find the container with id 41b7a4de3e1554dcbf3f737ac6a27223be383d86091ac421e08d194d1b061f80 Nov 02 23:12:52.181509 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:52.181460 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:12:52.181610 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:52.181551 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:12:52.412558 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:52.412514 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" event={"ID":"a4da1367-063f-4955-9313-26c9e7b97729","Type":"ContainerStarted","Data":"25de7f71386fefb8b10519afcf85d920550909664a601e38d01978930a28511e"} Nov 02 23:12:52.412752 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:52.412574 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" event={"ID":"a4da1367-063f-4955-9313-26c9e7b97729","Type":"ContainerStarted","Data":"41b7a4de3e1554dcbf3f737ac6a27223be383d86091ac421e08d194d1b061f80"} Nov 02 23:12:52.412752 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:52.412608 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:12:53.421505 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:53.421460 5114 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-zr4hl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:12:53.421920 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:53.421543 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" podUID="a4da1367-063f-4955-9313-26c9e7b97729" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:12:56.420876 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:56.420837 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:12:56.643723 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:12:56.643664 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" podStartSLOduration=5.643648605 podStartE2EDuration="5.643648605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:52.539647134 +0000 UTC m=+1447.892680509" watchObservedRunningTime="2026-11-02 23:12:56.643648605 +0000 UTC m=+1451.996681980" Nov 02 23:13:07.586742 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:13:07.586699 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:13:37.584754 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:13:37.584713 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:13:52.070965 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.070920 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld"] Nov 02 23:13:52.074843 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.074824 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.129925 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.129884 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ec2f9a-openshift-default-dockercfg-4nt5j\"" Nov 02 23:13:52.181050 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.181010 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.181238 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.181220 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlmpq\" (UniqueName: \"kubernetes.io/projected/1c9af20a-578a-4416-aee4-5fbdcb3370c4-kube-api-access-vlmpq\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.181329 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.181308 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.181393 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.181382 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.181528 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.181510 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.181627 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.181580 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.181627 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.181611 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.181737 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.181701 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.181789 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.181756 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.181947 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.181923 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld"] Nov 02 23:13:52.282777 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.282733 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.282777 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.282784 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.283041 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.282807 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.283041 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.282856 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.283041 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.282905 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.283041 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.282933 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.283244 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.283045 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlmpq\" (UniqueName: \"kubernetes.io/projected/1c9af20a-578a-4416-aee4-5fbdcb3370c4-kube-api-access-vlmpq\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.283244 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.283109 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.283244 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.283166 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.283244 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.283192 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.283244 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.283201 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.283615 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.283411 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.283615 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.283467 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.283770 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.283748 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.285676 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.285645 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.285867 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.285849 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.344176 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.344107 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.344315 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.344247 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlmpq\" (UniqueName: \"kubernetes.io/projected/1c9af20a-578a-4416-aee4-5fbdcb3370c4-kube-api-access-vlmpq\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.388610 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.388560 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:52.819191 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.819160 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld"] Nov 02 23:13:52.820877 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:13:52.820844 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9af20a_578a_4416_aee4_5fbdcb3370c4.slice/crio-7d6d097d605b8f5b4d46be70602c2e7ff4fdffe634961343ea31276143a24d0a WatchSource:0}: Error finding container 7d6d097d605b8f5b4d46be70602c2e7ff4fdffe634961343ea31276143a24d0a: Status 404 returned error can't find the container with id 7d6d097d605b8f5b4d46be70602c2e7ff4fdffe634961343ea31276143a24d0a Nov 02 23:13:52.823458 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.823423 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:13:52.823570 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.823519 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:13:52.823570 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:52.823548 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:13:53.837450 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:53.837409 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" event={"ID":"1c9af20a-578a-4416-aee4-5fbdcb3370c4","Type":"ContainerStarted","Data":"6248671ddcbd73b3791df7be705f0a7cdc98d4deb1246ce5c5d9e35e1ca193ce"} Nov 02 23:13:53.837971 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:53.837472 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" event={"ID":"1c9af20a-578a-4416-aee4-5fbdcb3370c4","Type":"ContainerStarted","Data":"7d6d097d605b8f5b4d46be70602c2e7ff4fdffe634961343ea31276143a24d0a"} Nov 02 23:13:54.085212 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:54.085144 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" podStartSLOduration=2.08512746 podStartE2EDuration="2.08512746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:13:54.034121748 +0000 UTC m=+1509.387155139" watchObservedRunningTime="2026-11-02 23:13:54.08512746 +0000 UTC m=+1509.438160835" Nov 02 23:13:54.389156 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:54.389114 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:13:54.390720 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:54.390697 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 23:13:54.390797 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:54.390746 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" podUID="1c9af20a-578a-4416-aee4-5fbdcb3370c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 23:13:55.393892 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:55.393848 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 23:13:55.394322 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:55.393908 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" podUID="1c9af20a-578a-4416-aee4-5fbdcb3370c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 23:13:56.389672 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:56.389632 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 23:13:56.389980 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:56.389952 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" podUID="1c9af20a-578a-4416-aee4-5fbdcb3370c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 23:13:57.389201 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:57.389166 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 23:13:57.389661 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:57.389231 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" podUID="1c9af20a-578a-4416-aee4-5fbdcb3370c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 23:13:58.389836 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:58.389787 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 23:13:58.390252 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:58.389854 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" podUID="1c9af20a-578a-4416-aee4-5fbdcb3370c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 23:13:59.389834 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:59.389796 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 23:13:59.390270 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:13:59.389866 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" podUID="1c9af20a-578a-4416-aee4-5fbdcb3370c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 23:14:00.393660 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:14:00.393631 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:14:00.394112 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:14:00.394091 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:14:00.394907 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:14:00.394885 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:14:07.584946 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:14:07.584906 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:14:37.588770 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:14:37.588677 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:15:00.178853 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.178811 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz"] Nov 02 23:15:00.183343 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.183316 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz" Nov 02 23:15:00.200139 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.200107 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26"] Nov 02 23:15:00.204508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.204467 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7"] Nov 02 23:15:00.204645 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.204571 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26" Nov 02 23:15:00.208171 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.208146 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7" Nov 02 23:15:00.228170 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.228142 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5"] Nov 02 23:15:00.231779 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.231760 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5" Nov 02 23:15:00.246835 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.246805 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:15:00.274272 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.274112 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1bf336-e967-4582-83b8-12584603fd63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5\" (UID: \"3a1bf336-e967-4582-83b8-12584603fd63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5" Nov 02 23:15:00.274272 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.274158 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqnvf\" (UniqueName: \"kubernetes.io/projected/3a1bf336-e967-4582-83b8-12584603fd63-kube-api-access-sqnvf\") pod \"maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5\" (UID: \"3a1bf336-e967-4582-83b8-12584603fd63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5" Nov 02 23:15:00.274272 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.274197 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w99w9\" (UniqueName: \"kubernetes.io/projected/83a4dc67-3b2b-463f-a869-76c7d9b72195-kube-api-access-w99w9\") pod \"maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7\" (UID: \"83a4dc67-3b2b-463f-a869-76c7d9b72195\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7" Nov 02 23:15:00.274272 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.274308 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gddl9\" (UniqueName: \"kubernetes.io/projected/6b3a4d68-657c-4da3-8fbf-891f0926b134-kube-api-access-gddl9\") pod \"maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz\" (UID: \"6b3a4d68-657c-4da3-8fbf-891f0926b134\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz" Nov 02 23:15:00.274272 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.274420 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a4dc67-3b2b-463f-a869-76c7d9b72195-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7\" (UID: \"83a4dc67-3b2b-463f-a869-76c7d9b72195\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7" Nov 02 23:15:00.274272 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.274456 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wmv4\" (UniqueName: \"kubernetes.io/projected/290360eb-55b0-42b3-9171-c74f9bb33bac-kube-api-access-7wmv4\") pod \"maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26\" (UID: \"290360eb-55b0-42b3-9171-c74f9bb33bac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26" Nov 02 23:15:00.274272 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.274511 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/290360eb-55b0-42b3-9171-c74f9bb33bac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26\" (UID: \"290360eb-55b0-42b3-9171-c74f9bb33bac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26" Nov 02 23:15:00.274272 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.274589 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3a4d68-657c-4da3-8fbf-891f0926b134-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz\" (UID: \"6b3a4d68-657c-4da3-8fbf-891f0926b134\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz" Nov 02 23:15:00.283053 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.283020 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz"] Nov 02 23:15:00.334535 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.332574 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5"] Nov 02 23:15:00.334730 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.334610 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26"] Nov 02 23:15:00.375851 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.375814 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a4dc67-3b2b-463f-a869-76c7d9b72195-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7\" (UID: \"83a4dc67-3b2b-463f-a869-76c7d9b72195\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7" Nov 02 23:15:00.376046 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.375861 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wmv4\" (UniqueName: \"kubernetes.io/projected/290360eb-55b0-42b3-9171-c74f9bb33bac-kube-api-access-7wmv4\") pod \"maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26\" (UID: \"290360eb-55b0-42b3-9171-c74f9bb33bac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26" Nov 02 23:15:00.376046 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.375976 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/290360eb-55b0-42b3-9171-c74f9bb33bac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26\" (UID: \"290360eb-55b0-42b3-9171-c74f9bb33bac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26" Nov 02 23:15:00.376163 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.376052 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3a4d68-657c-4da3-8fbf-891f0926b134-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz\" (UID: \"6b3a4d68-657c-4da3-8fbf-891f0926b134\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz" Nov 02 23:15:00.376271 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.376236 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1bf336-e967-4582-83b8-12584603fd63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5\" (UID: \"3a1bf336-e967-4582-83b8-12584603fd63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5" Nov 02 23:15:00.376271 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.376270 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqnvf\" (UniqueName: \"kubernetes.io/projected/3a1bf336-e967-4582-83b8-12584603fd63-kube-api-access-sqnvf\") pod \"maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5\" (UID: \"3a1bf336-e967-4582-83b8-12584603fd63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5" Nov 02 23:15:00.376469 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.376304 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w99w9\" (UniqueName: \"kubernetes.io/projected/83a4dc67-3b2b-463f-a869-76c7d9b72195-kube-api-access-w99w9\") pod \"maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7\" (UID: \"83a4dc67-3b2b-463f-a869-76c7d9b72195\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7" Nov 02 23:15:00.376469 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.376412 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gddl9\" (UniqueName: \"kubernetes.io/projected/6b3a4d68-657c-4da3-8fbf-891f0926b134-kube-api-access-gddl9\") pod \"maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz\" (UID: \"6b3a4d68-657c-4da3-8fbf-891f0926b134\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz" Nov 02 23:15:00.376810 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.376780 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a4dc67-3b2b-463f-a869-76c7d9b72195-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7\" (UID: \"83a4dc67-3b2b-463f-a869-76c7d9b72195\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7" Nov 02 23:15:00.376929 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.376851 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3a4d68-657c-4da3-8fbf-891f0926b134-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz\" (UID: \"6b3a4d68-657c-4da3-8fbf-891f0926b134\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz" Nov 02 23:15:00.377512 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.377447 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1bf336-e967-4582-83b8-12584603fd63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5\" (UID: \"3a1bf336-e967-4582-83b8-12584603fd63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5" Nov 02 23:15:00.382352 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.382313 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7"] Nov 02 23:15:00.382552 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.382398 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/290360eb-55b0-42b3-9171-c74f9bb33bac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26\" (UID: \"290360eb-55b0-42b3-9171-c74f9bb33bac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26" Nov 02 23:15:00.436963 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.436880 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wmv4\" (UniqueName: \"kubernetes.io/projected/290360eb-55b0-42b3-9171-c74f9bb33bac-kube-api-access-7wmv4\") pod \"maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26\" (UID: \"290360eb-55b0-42b3-9171-c74f9bb33bac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26" Nov 02 23:15:00.436963 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.436947 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gddl9\" (UniqueName: \"kubernetes.io/projected/6b3a4d68-657c-4da3-8fbf-891f0926b134-kube-api-access-gddl9\") pod \"maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz\" (UID: \"6b3a4d68-657c-4da3-8fbf-891f0926b134\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz" Nov 02 23:15:00.437149 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.437130 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w99w9\" (UniqueName: \"kubernetes.io/projected/83a4dc67-3b2b-463f-a869-76c7d9b72195-kube-api-access-w99w9\") pod \"maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7\" (UID: \"83a4dc67-3b2b-463f-a869-76c7d9b72195\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7" Nov 02 23:15:00.437190 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.437134 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqnvf\" (UniqueName: \"kubernetes.io/projected/3a1bf336-e967-4582-83b8-12584603fd63-kube-api-access-sqnvf\") pod \"maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5\" (UID: \"3a1bf336-e967-4582-83b8-12584603fd63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5" Nov 02 23:15:00.497573 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.497530 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz" Nov 02 23:15:00.522330 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.522295 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26" Nov 02 23:15:00.529020 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.528975 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7" Nov 02 23:15:00.545632 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.545592 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5" Nov 02 23:15:00.884892 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.884865 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz"] Nov 02 23:15:00.886181 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:15:00.886158 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3a4d68_657c_4da3_8fbf_891f0926b134.slice/crio-e02b93076df2f9f66afeeb30d5fb581c794ec4871da3e623e0fc2f17e8d1b5b4 WatchSource:0}: Error finding container e02b93076df2f9f66afeeb30d5fb581c794ec4871da3e623e0fc2f17e8d1b5b4: Status 404 returned error can't find the container with id e02b93076df2f9f66afeeb30d5fb581c794ec4871da3e623e0fc2f17e8d1b5b4 Nov 02 23:15:00.888172 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.888153 5114 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:15:00.935458 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.935425 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26"] Nov 02 23:15:00.937121 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:15:00.937085 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod290360eb_55b0_42b3_9171_c74f9bb33bac.slice/crio-d783a6aae4f05916b8ad4a3b10e7dc06ad4e2e19c234b13043dea02f9f4bfe9a WatchSource:0}: Error finding container d783a6aae4f05916b8ad4a3b10e7dc06ad4e2e19c234b13043dea02f9f4bfe9a: Status 404 returned error can't find the container with id d783a6aae4f05916b8ad4a3b10e7dc06ad4e2e19c234b13043dea02f9f4bfe9a Nov 02 23:15:00.979553 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.979520 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7"] Nov 02 23:15:00.981795 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:15:00.981762 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a4dc67_3b2b_463f_a869_76c7d9b72195.slice/crio-b7e3ae042ae95648c6266b2abfe305014a90b56413db76957b929667656a3800 WatchSource:0}: Error finding container b7e3ae042ae95648c6266b2abfe305014a90b56413db76957b929667656a3800: Status 404 returned error can't find the container with id b7e3ae042ae95648c6266b2abfe305014a90b56413db76957b929667656a3800 Nov 02 23:15:00.988618 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:00.988586 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5"] Nov 02 23:15:00.989437 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:15:00.989413 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1bf336_e967_4582_83b8_12584603fd63.slice/crio-22f76864129297d89ce47a9a92ff655615d36889883c9be4c6b149f25caca89d WatchSource:0}: Error finding container 22f76864129297d89ce47a9a92ff655615d36889883c9be4c6b149f25caca89d: Status 404 returned error can't find the container with id 22f76864129297d89ce47a9a92ff655615d36889883c9be4c6b149f25caca89d Nov 02 23:15:01.274938 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:01.274834 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5" event={"ID":"3a1bf336-e967-4582-83b8-12584603fd63","Type":"ContainerStarted","Data":"22f76864129297d89ce47a9a92ff655615d36889883c9be4c6b149f25caca89d"} Nov 02 23:15:01.276250 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:01.276214 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz" event={"ID":"6b3a4d68-657c-4da3-8fbf-891f0926b134","Type":"ContainerStarted","Data":"e02b93076df2f9f66afeeb30d5fb581c794ec4871da3e623e0fc2f17e8d1b5b4"} Nov 02 23:15:01.277409 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:01.277385 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7" event={"ID":"83a4dc67-3b2b-463f-a869-76c7d9b72195","Type":"ContainerStarted","Data":"b7e3ae042ae95648c6266b2abfe305014a90b56413db76957b929667656a3800"} Nov 02 23:15:01.278662 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:01.278624 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26" event={"ID":"290360eb-55b0-42b3-9171-c74f9bb33bac","Type":"ContainerStarted","Data":"d783a6aae4f05916b8ad4a3b10e7dc06ad4e2e19c234b13043dea02f9f4bfe9a"} Nov 02 23:15:03.630264 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:03.630232 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dea36f3-f4r92"] Nov 02 23:15:03.634601 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:03.634578 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dea36f3-f4r92" Nov 02 23:15:03.688896 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:03.688853 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:15:03.694228 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:03.694194 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dea36f3-f4r92"] Nov 02 23:15:03.723088 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:03.723048 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033ac52a-46b6-4b42-b9c0-b5f31a6b1675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dea36f3-f4r92\" (UID: \"033ac52a-46b6-4b42-b9c0-b5f31a6b1675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dea36f3-f4r92" Nov 02 23:15:03.723323 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:03.723098 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtzgp\" (UniqueName: \"kubernetes.io/projected/033ac52a-46b6-4b42-b9c0-b5f31a6b1675-kube-api-access-rtzgp\") pod \"maas-api-revoke-sub-6dea36f3-f4r92\" (UID: \"033ac52a-46b6-4b42-b9c0-b5f31a6b1675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dea36f3-f4r92" Nov 02 23:15:03.723323 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:03.723131 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033ac52a-46b6-4b42-b9c0-b5f31a6b1675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dea36f3-f4r92\" (UID: \"033ac52a-46b6-4b42-b9c0-b5f31a6b1675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dea36f3-f4r92" Nov 02 23:15:03.824214 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:03.824169 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033ac52a-46b6-4b42-b9c0-b5f31a6b1675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dea36f3-f4r92\" (UID: \"033ac52a-46b6-4b42-b9c0-b5f31a6b1675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dea36f3-f4r92" Nov 02 23:15:03.824421 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:03.824226 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtzgp\" (UniqueName: \"kubernetes.io/projected/033ac52a-46b6-4b42-b9c0-b5f31a6b1675-kube-api-access-rtzgp\") pod \"maas-api-revoke-sub-6dea36f3-f4r92\" (UID: \"033ac52a-46b6-4b42-b9c0-b5f31a6b1675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dea36f3-f4r92" Nov 02 23:15:03.824421 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:03.824252 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033ac52a-46b6-4b42-b9c0-b5f31a6b1675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dea36f3-f4r92\" (UID: \"033ac52a-46b6-4b42-b9c0-b5f31a6b1675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dea36f3-f4r92" Nov 02 23:15:03.825128 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:03.825101 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033ac52a-46b6-4b42-b9c0-b5f31a6b1675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dea36f3-f4r92\" (UID: \"033ac52a-46b6-4b42-b9c0-b5f31a6b1675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dea36f3-f4r92" Nov 02 23:15:03.827414 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:03.827375 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033ac52a-46b6-4b42-b9c0-b5f31a6b1675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dea36f3-f4r92\" (UID: \"033ac52a-46b6-4b42-b9c0-b5f31a6b1675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dea36f3-f4r92" Nov 02 23:15:03.886150 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:03.886076 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtzgp\" (UniqueName: \"kubernetes.io/projected/033ac52a-46b6-4b42-b9c0-b5f31a6b1675-kube-api-access-rtzgp\") pod \"maas-api-revoke-sub-6dea36f3-f4r92\" (UID: \"033ac52a-46b6-4b42-b9c0-b5f31a6b1675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dea36f3-f4r92" Nov 02 23:15:03.950464 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:03.950420 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dea36f3-f4r92" Nov 02 23:15:04.348166 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:04.348120 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dea36f3-f4r92"] Nov 02 23:15:04.354665 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:15:04.354613 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033ac52a_46b6_4b42_b9c0_b5f31a6b1675.slice/crio-e8099ab338e1841bbaf7f5ccf3ef4aacd156372abb79ba3a758623382596299e WatchSource:0}: Error finding container e8099ab338e1841bbaf7f5ccf3ef4aacd156372abb79ba3a758623382596299e: Status 404 returned error can't find the container with id e8099ab338e1841bbaf7f5ccf3ef4aacd156372abb79ba3a758623382596299e Nov 02 23:15:05.316313 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:05.316268 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dea36f3-f4r92" event={"ID":"033ac52a-46b6-4b42-b9c0-b5f31a6b1675","Type":"ContainerStarted","Data":"e8099ab338e1841bbaf7f5ccf3ef4aacd156372abb79ba3a758623382596299e"} Nov 02 23:15:07.332150 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:07.332108 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7" event={"ID":"83a4dc67-3b2b-463f-a869-76c7d9b72195","Type":"ContainerStarted","Data":"5cbe5b656090cd86d788f9b9d46afc3573e08d9ed50ff4efeef85b5f32739e7a"} Nov 02 23:15:07.333860 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:07.333821 5114 generic.go:356] "Generic (PLEG): container finished" podID="290360eb-55b0-42b3-9171-c74f9bb33bac" containerID="d94bff5295b94484911f737eb0dc9ad3c62e9adef7196f45f780114f95a2eb3c" exitCode=60 Nov 02 23:15:07.333994 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:07.333954 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26" event={"ID":"290360eb-55b0-42b3-9171-c74f9bb33bac","Type":"ContainerDied","Data":"d94bff5295b94484911f737eb0dc9ad3c62e9adef7196f45f780114f95a2eb3c"} Nov 02 23:15:07.334182 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:07.334162 5114 scope.go:122] "RemoveContainer" containerID="d94bff5295b94484911f737eb0dc9ad3c62e9adef7196f45f780114f95a2eb3c" Nov 02 23:15:07.335969 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:07.335815 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dea36f3-f4r92" event={"ID":"033ac52a-46b6-4b42-b9c0-b5f31a6b1675","Type":"ContainerStarted","Data":"6c2a94a88907463d9ba019cd5e1acd3a783faa6322ac95ffc29a40eca585ed3b"} Nov 02 23:15:07.337417 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:07.337391 5114 generic.go:356] "Generic (PLEG): container finished" podID="3a1bf336-e967-4582-83b8-12584603fd63" containerID="815e890a2763e33582ef109305690b12fba8593c26b279e6e16a668a06e469e5" exitCode=60 Nov 02 23:15:07.337571 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:07.337526 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5" event={"ID":"3a1bf336-e967-4582-83b8-12584603fd63","Type":"ContainerDied","Data":"815e890a2763e33582ef109305690b12fba8593c26b279e6e16a668a06e469e5"} Nov 02 23:15:07.337801 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:07.337788 5114 scope.go:122] "RemoveContainer" containerID="815e890a2763e33582ef109305690b12fba8593c26b279e6e16a668a06e469e5" Nov 02 23:15:07.339209 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:07.339187 5114 generic.go:356] "Generic (PLEG): container finished" podID="6b3a4d68-657c-4da3-8fbf-891f0926b134" containerID="fce1cd1aac8adc8667d072b7a38d8310a234fcec9a03af1269d5fe90e143a208" exitCode=60 Nov 02 23:15:07.339308 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:07.339217 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz" event={"ID":"6b3a4d68-657c-4da3-8fbf-891f0926b134","Type":"ContainerDied","Data":"fce1cd1aac8adc8667d072b7a38d8310a234fcec9a03af1269d5fe90e143a208"} Nov 02 23:15:07.339553 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:07.339532 5114 scope.go:122] "RemoveContainer" containerID="fce1cd1aac8adc8667d072b7a38d8310a234fcec9a03af1269d5fe90e143a208" Nov 02 23:15:07.447772 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:07.447709 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7" podStartSLOduration=1.865980985 podStartE2EDuration="7.447691979s" totalImagesPullingTime="5.581710994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:07.446293058 +0000 UTC m=+1582.799326436" watchObservedRunningTime="2026-11-02 23:15:07.447691979 +0000 UTC m=+1582.800725355" Nov 02 23:15:07.676582 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:07.676475 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dea36f3-f4r92" podStartSLOduration=2.478721562 podStartE2EDuration="4.676458806s" totalImagesPullingTime="2.197737244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:07.674828585 +0000 UTC m=+1583.027861977" watchObservedRunningTime="2026-11-02 23:15:07.676458806 +0000 UTC m=+1583.029492180" Nov 02 23:15:07.709717 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:15:07.709671 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:15:08.347342 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:08.347306 5114 generic.go:356] "Generic (PLEG): container finished" podID="83a4dc67-3b2b-463f-a869-76c7d9b72195" containerID="5cbe5b656090cd86d788f9b9d46afc3573e08d9ed50ff4efeef85b5f32739e7a" exitCode=60 Nov 02 23:15:08.347838 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:08.347599 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7" event={"ID":"83a4dc67-3b2b-463f-a869-76c7d9b72195","Type":"ContainerDied","Data":"5cbe5b656090cd86d788f9b9d46afc3573e08d9ed50ff4efeef85b5f32739e7a"} Nov 02 23:15:08.347838 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:08.347778 5114 scope.go:122] "RemoveContainer" containerID="5cbe5b656090cd86d788f9b9d46afc3573e08d9ed50ff4efeef85b5f32739e7a" Nov 02 23:15:08.352671 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:08.352629 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26" event={"ID":"290360eb-55b0-42b3-9171-c74f9bb33bac","Type":"ContainerStarted","Data":"e289558c05f17813eaca688d97abb88d74aa340c5d29389e2da93c1c06071e92"} Nov 02 23:15:08.355315 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:08.355275 5114 generic.go:356] "Generic (PLEG): container finished" podID="033ac52a-46b6-4b42-b9c0-b5f31a6b1675" containerID="6c2a94a88907463d9ba019cd5e1acd3a783faa6322ac95ffc29a40eca585ed3b" exitCode=0 Nov 02 23:15:08.355557 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:08.355531 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dea36f3-f4r92" event={"ID":"033ac52a-46b6-4b42-b9c0-b5f31a6b1675","Type":"ContainerDied","Data":"6c2a94a88907463d9ba019cd5e1acd3a783faa6322ac95ffc29a40eca585ed3b"} Nov 02 23:15:08.358035 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:08.357997 5114 generic.go:356] "Generic (PLEG): container finished" podID="3a1bf336-e967-4582-83b8-12584603fd63" containerID="b48cff6181ee5efc9ac7c59e93e48239bd0d7a9c719e8db74db9a603a4519c57" exitCode=60 Nov 02 23:15:08.358200 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:08.358054 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5" event={"ID":"3a1bf336-e967-4582-83b8-12584603fd63","Type":"ContainerDied","Data":"b48cff6181ee5efc9ac7c59e93e48239bd0d7a9c719e8db74db9a603a4519c57"} Nov 02 23:15:08.358200 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:08.358131 5114 scope.go:122] "RemoveContainer" containerID="815e890a2763e33582ef109305690b12fba8593c26b279e6e16a668a06e469e5" Nov 02 23:15:08.358533 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:08.358468 5114 scope.go:122] "RemoveContainer" containerID="b48cff6181ee5efc9ac7c59e93e48239bd0d7a9c719e8db74db9a603a4519c57" Nov 02 23:15:08.358918 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:15:08.358875 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5_odh-ai-gateway-infra(3a1bf336-e967-4582-83b8-12584603fd63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5" podUID="3a1bf336-e967-4582-83b8-12584603fd63" Nov 02 23:15:08.361014 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:08.360980 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz" event={"ID":"6b3a4d68-657c-4da3-8fbf-891f0926b134","Type":"ContainerStarted","Data":"dee0c0754de46a48fdfa71000209ffc4d18f088c176cdc85de8b44e6fa2008b7"} Nov 02 23:15:08.808641 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:08.808586 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26" podStartSLOduration=3.741375558 podStartE2EDuration="8.808569973s" totalImagesPullingTime="5.067194415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:08.807003652 +0000 UTC m=+1584.160037028" watchObservedRunningTime="2026-11-02 23:15:08.808569973 +0000 UTC m=+1584.161603402" Nov 02 23:15:08.923744 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:08.923626 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz" podStartSLOduration=3.391288774 podStartE2EDuration="8.923609321s" totalImagesPullingTime="5.532320547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:08.921264581 +0000 UTC m=+1584.274297976" watchObservedRunningTime="2026-11-02 23:15:08.923609321 +0000 UTC m=+1584.276642690" Nov 02 23:15:09.369392 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:09.369356 5114 generic.go:356] "Generic (PLEG): container finished" podID="6b3a4d68-657c-4da3-8fbf-891f0926b134" containerID="dee0c0754de46a48fdfa71000209ffc4d18f088c176cdc85de8b44e6fa2008b7" exitCode=60 Nov 02 23:15:09.369937 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:09.369440 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz" event={"ID":"6b3a4d68-657c-4da3-8fbf-891f0926b134","Type":"ContainerDied","Data":"dee0c0754de46a48fdfa71000209ffc4d18f088c176cdc85de8b44e6fa2008b7"} Nov 02 23:15:09.369937 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:09.369550 5114 scope.go:122] "RemoveContainer" containerID="fce1cd1aac8adc8667d072b7a38d8310a234fcec9a03af1269d5fe90e143a208" Nov 02 23:15:09.369937 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:09.369817 5114 scope.go:122] "RemoveContainer" containerID="dee0c0754de46a48fdfa71000209ffc4d18f088c176cdc85de8b44e6fa2008b7" Nov 02 23:15:09.370201 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:15:09.370163 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz_odh-ai-gateway-infra(6b3a4d68-657c-4da3-8fbf-891f0926b134)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz" podUID="6b3a4d68-657c-4da3-8fbf-891f0926b134" Nov 02 23:15:09.371648 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:09.371626 5114 generic.go:356] "Generic (PLEG): container finished" podID="83a4dc67-3b2b-463f-a869-76c7d9b72195" containerID="2126f7a4eaf242ccd52f4abff716df9250b975301d772fd6ff1c8ae5b2a446c2" exitCode=60 Nov 02 23:15:09.371751 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:09.371707 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7" event={"ID":"83a4dc67-3b2b-463f-a869-76c7d9b72195","Type":"ContainerDied","Data":"2126f7a4eaf242ccd52f4abff716df9250b975301d772fd6ff1c8ae5b2a446c2"} Nov 02 23:15:09.372057 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:09.372040 5114 scope.go:122] "RemoveContainer" containerID="2126f7a4eaf242ccd52f4abff716df9250b975301d772fd6ff1c8ae5b2a446c2" Nov 02 23:15:09.372370 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:15:09.372353 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7_odh-ai-gateway-infra(83a4dc67-3b2b-463f-a869-76c7d9b72195)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7" podUID="83a4dc67-3b2b-463f-a869-76c7d9b72195" Nov 02 23:15:09.373575 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:09.373548 5114 generic.go:356] "Generic (PLEG): container finished" podID="290360eb-55b0-42b3-9171-c74f9bb33bac" containerID="e289558c05f17813eaca688d97abb88d74aa340c5d29389e2da93c1c06071e92" exitCode=60 Nov 02 23:15:09.373641 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:09.373625 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26" event={"ID":"290360eb-55b0-42b3-9171-c74f9bb33bac","Type":"ContainerDied","Data":"e289558c05f17813eaca688d97abb88d74aa340c5d29389e2da93c1c06071e92"} Nov 02 23:15:09.373927 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:09.373914 5114 scope.go:122] "RemoveContainer" containerID="e289558c05f17813eaca688d97abb88d74aa340c5d29389e2da93c1c06071e92" Nov 02 23:15:09.374220 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:15:09.374196 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26_odh-ai-gateway-infra(290360eb-55b0-42b3-9171-c74f9bb33bac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26" podUID="290360eb-55b0-42b3-9171-c74f9bb33bac" Nov 02 23:15:09.375922 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:09.375904 5114 scope.go:122] "RemoveContainer" containerID="b48cff6181ee5efc9ac7c59e93e48239bd0d7a9c719e8db74db9a603a4519c57" Nov 02 23:15:09.376279 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:15:09.376256 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5_odh-ai-gateway-infra(3a1bf336-e967-4582-83b8-12584603fd63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5" podUID="3a1bf336-e967-4582-83b8-12584603fd63" Nov 02 23:15:09.382377 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:09.382353 5114 scope.go:122] "RemoveContainer" containerID="5cbe5b656090cd86d788f9b9d46afc3573e08d9ed50ff4efeef85b5f32739e7a" Nov 02 23:15:09.397126 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:09.397094 5114 scope.go:122] "RemoveContainer" containerID="d94bff5295b94484911f737eb0dc9ad3c62e9adef7196f45f780114f95a2eb3c" Nov 02 23:15:09.533286 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:09.533249 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dea36f3-f4r92" Nov 02 23:15:09.609525 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:09.609467 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/033ac52a-46b6-4b42-b9c0-b5f31a6b1675-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033ac52a-46b6-4b42-b9c0-b5f31a6b1675-maas-api-xks-ca\") pod \"033ac52a-46b6-4b42-b9c0-b5f31a6b1675\" (UID: \"033ac52a-46b6-4b42-b9c0-b5f31a6b1675\") " Nov 02 23:15:09.609703 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:09.609661 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/033ac52a-46b6-4b42-b9c0-b5f31a6b1675-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033ac52a-46b6-4b42-b9c0-b5f31a6b1675-maas-api-openshift-ca\") pod \"033ac52a-46b6-4b42-b9c0-b5f31a6b1675\" (UID: \"033ac52a-46b6-4b42-b9c0-b5f31a6b1675\") " Nov 02 23:15:09.609744 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:09.609708 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/033ac52a-46b6-4b42-b9c0-b5f31a6b1675-kube-api-access-rtzgp\" (UniqueName: \"kubernetes.io/projected/033ac52a-46b6-4b42-b9c0-b5f31a6b1675-kube-api-access-rtzgp\") pod \"033ac52a-46b6-4b42-b9c0-b5f31a6b1675\" (UID: \"033ac52a-46b6-4b42-b9c0-b5f31a6b1675\") " Nov 02 23:15:09.610043 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:09.610008 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033ac52a-46b6-4b42-b9c0-b5f31a6b1675-maas-api-openshift-ca" pod "033ac52a-46b6-4b42-b9c0-b5f31a6b1675" (UID: "033ac52a-46b6-4b42-b9c0-b5f31a6b1675"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:15:09.610311 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:09.610294 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033ac52a-46b6-4b42-b9c0-b5f31a6b1675-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:09.611812 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:09.611785 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/033ac52a-46b6-4b42-b9c0-b5f31a6b1675-maas-api-xks-ca" pod "033ac52a-46b6-4b42-b9c0-b5f31a6b1675" (UID: "033ac52a-46b6-4b42-b9c0-b5f31a6b1675"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:15:09.611966 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:09.611946 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033ac52a-46b6-4b42-b9c0-b5f31a6b1675-kube-api-access-rtzgp" pod "033ac52a-46b6-4b42-b9c0-b5f31a6b1675" (UID: "033ac52a-46b6-4b42-b9c0-b5f31a6b1675"). InnerVolumeSpecName "kube-api-access-rtzgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:15:09.712204 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:09.712163 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtzgp\" (UniqueName: \"kubernetes.io/projected/033ac52a-46b6-4b42-b9c0-b5f31a6b1675-kube-api-access-rtzgp\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:09.712204 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:09.712199 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033ac52a-46b6-4b42-b9c0-b5f31a6b1675-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:10.382999 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:10.382971 5114 scope.go:122] "RemoveContainer" containerID="dee0c0754de46a48fdfa71000209ffc4d18f088c176cdc85de8b44e6fa2008b7" Nov 02 23:15:10.383459 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:15:10.383278 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz_odh-ai-gateway-infra(6b3a4d68-657c-4da3-8fbf-891f0926b134)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz" podUID="6b3a4d68-657c-4da3-8fbf-891f0926b134" Nov 02 23:15:10.384759 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:10.384723 5114 scope.go:122] "RemoveContainer" containerID="2126f7a4eaf242ccd52f4abff716df9250b975301d772fd6ff1c8ae5b2a446c2" Nov 02 23:15:10.385037 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:15:10.385016 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7_odh-ai-gateway-infra(83a4dc67-3b2b-463f-a869-76c7d9b72195)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7" podUID="83a4dc67-3b2b-463f-a869-76c7d9b72195" Nov 02 23:15:10.386310 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:10.386289 5114 scope.go:122] "RemoveContainer" containerID="e289558c05f17813eaca688d97abb88d74aa340c5d29389e2da93c1c06071e92" Nov 02 23:15:10.386630 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:15:10.386592 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26_odh-ai-gateway-infra(290360eb-55b0-42b3-9171-c74f9bb33bac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26" podUID="290360eb-55b0-42b3-9171-c74f9bb33bac" Nov 02 23:15:10.387425 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:10.387393 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dea36f3-f4r92" event={"ID":"033ac52a-46b6-4b42-b9c0-b5f31a6b1675","Type":"ContainerDied","Data":"e8099ab338e1841bbaf7f5ccf3ef4aacd156372abb79ba3a758623382596299e"} Nov 02 23:15:10.387425 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:10.387408 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dea36f3-f4r92" Nov 02 23:15:10.387568 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:10.387444 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8099ab338e1841bbaf7f5ccf3ef4aacd156372abb79ba3a758623382596299e" Nov 02 23:15:19.786773 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:19.786740 5114 scope.go:122] "RemoveContainer" containerID="e289558c05f17813eaca688d97abb88d74aa340c5d29389e2da93c1c06071e92" Nov 02 23:15:20.453603 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:20.453511 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26" event={"ID":"290360eb-55b0-42b3-9171-c74f9bb33bac","Type":"ContainerStarted","Data":"f6d4edf6c233499d7d17122be2c95ad556c40900048eb9c7226729256f4bf067"} Nov 02 23:15:21.461582 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:21.461541 5114 generic.go:356] "Generic (PLEG): container finished" podID="290360eb-55b0-42b3-9171-c74f9bb33bac" containerID="f6d4edf6c233499d7d17122be2c95ad556c40900048eb9c7226729256f4bf067" exitCode=60 Nov 02 23:15:21.462046 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:21.461613 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26" event={"ID":"290360eb-55b0-42b3-9171-c74f9bb33bac","Type":"ContainerDied","Data":"f6d4edf6c233499d7d17122be2c95ad556c40900048eb9c7226729256f4bf067"} Nov 02 23:15:21.462046 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:21.461674 5114 scope.go:122] "RemoveContainer" containerID="e289558c05f17813eaca688d97abb88d74aa340c5d29389e2da93c1c06071e92" Nov 02 23:15:21.462172 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:21.462084 5114 scope.go:122] "RemoveContainer" containerID="f6d4edf6c233499d7d17122be2c95ad556c40900048eb9c7226729256f4bf067" Nov 02 23:15:21.462447 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:15:21.462417 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26_odh-ai-gateway-infra(290360eb-55b0-42b3-9171-c74f9bb33bac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26" podUID="290360eb-55b0-42b3-9171-c74f9bb33bac" Nov 02 23:15:21.577825 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:21.577792 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26"] Nov 02 23:15:21.787224 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:21.787184 5114 scope.go:122] "RemoveContainer" containerID="dee0c0754de46a48fdfa71000209ffc4d18f088c176cdc85de8b44e6fa2008b7" Nov 02 23:15:21.787441 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:21.787350 5114 scope.go:122] "RemoveContainer" containerID="b48cff6181ee5efc9ac7c59e93e48239bd0d7a9c719e8db74db9a603a4519c57" Nov 02 23:15:22.469383 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:22.469280 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5" event={"ID":"3a1bf336-e967-4582-83b8-12584603fd63","Type":"ContainerStarted","Data":"4d7b0f0fdafc596b5188226c815ed4187dd714b29460fbad6c716f2f3519c6a8"} Nov 02 23:15:22.471345 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:22.471307 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz" event={"ID":"6b3a4d68-657c-4da3-8fbf-891f0926b134","Type":"ContainerStarted","Data":"7949375b59b23838ead5ec3e8ad9e46a036421b2139dc20f476a790f90a2dfb7"} Nov 02 23:15:22.589177 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:22.589105 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5" podStartSLOduration=17.2012597 podStartE2EDuration="22.589081942s" totalImagesPullingTime="5.387822242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:22.583981118 +0000 UTC m=+1597.937014493" watchObservedRunningTime="2026-11-02 23:15:22.589081942 +0000 UTC m=+1597.942115318" Nov 02 23:15:22.633536 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:22.633504 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26" Nov 02 23:15:22.691378 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:22.691342 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/290360eb-55b0-42b3-9171-c74f9bb33bac-kube-api-access-7wmv4\" (UniqueName: \"kubernetes.io/projected/290360eb-55b0-42b3-9171-c74f9bb33bac-kube-api-access-7wmv4\") pod \"290360eb-55b0-42b3-9171-c74f9bb33bac\" (UID: \"290360eb-55b0-42b3-9171-c74f9bb33bac\") " Nov 02 23:15:22.691586 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:22.691413 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/290360eb-55b0-42b3-9171-c74f9bb33bac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/290360eb-55b0-42b3-9171-c74f9bb33bac-maas-api-service-ca\") pod \"290360eb-55b0-42b3-9171-c74f9bb33bac\" (UID: \"290360eb-55b0-42b3-9171-c74f9bb33bac\") " Nov 02 23:15:22.691800 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:22.691779 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/290360eb-55b0-42b3-9171-c74f9bb33bac-maas-api-service-ca" pod "290360eb-55b0-42b3-9171-c74f9bb33bac" (UID: "290360eb-55b0-42b3-9171-c74f9bb33bac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:15:22.691979 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:22.691951 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/290360eb-55b0-42b3-9171-c74f9bb33bac-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:22.693775 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:22.693749 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290360eb-55b0-42b3-9171-c74f9bb33bac-kube-api-access-7wmv4" pod "290360eb-55b0-42b3-9171-c74f9bb33bac" (UID: "290360eb-55b0-42b3-9171-c74f9bb33bac"). InnerVolumeSpecName "kube-api-access-7wmv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:15:22.787568 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:22.787521 5114 scope.go:122] "RemoveContainer" containerID="2126f7a4eaf242ccd52f4abff716df9250b975301d772fd6ff1c8ae5b2a446c2" Nov 02 23:15:22.794819 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:22.793177 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wmv4\" (UniqueName: \"kubernetes.io/projected/290360eb-55b0-42b3-9171-c74f9bb33bac-kube-api-access-7wmv4\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:23.481147 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:23.481063 5114 generic.go:356] "Generic (PLEG): container finished" podID="3a1bf336-e967-4582-83b8-12584603fd63" containerID="4d7b0f0fdafc596b5188226c815ed4187dd714b29460fbad6c716f2f3519c6a8" exitCode=60 Nov 02 23:15:23.481147 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:23.481135 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5" event={"ID":"3a1bf336-e967-4582-83b8-12584603fd63","Type":"ContainerDied","Data":"4d7b0f0fdafc596b5188226c815ed4187dd714b29460fbad6c716f2f3519c6a8"} Nov 02 23:15:23.481661 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:23.481198 5114 scope.go:122] "RemoveContainer" containerID="b48cff6181ee5efc9ac7c59e93e48239bd0d7a9c719e8db74db9a603a4519c57" Nov 02 23:15:23.481661 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:23.481614 5114 scope.go:122] "RemoveContainer" containerID="4d7b0f0fdafc596b5188226c815ed4187dd714b29460fbad6c716f2f3519c6a8" Nov 02 23:15:23.482020 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:15:23.481992 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5_odh-ai-gateway-infra(3a1bf336-e967-4582-83b8-12584603fd63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5" podUID="3a1bf336-e967-4582-83b8-12584603fd63" Nov 02 23:15:23.483533 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:23.483473 5114 generic.go:356] "Generic (PLEG): container finished" podID="6b3a4d68-657c-4da3-8fbf-891f0926b134" containerID="7949375b59b23838ead5ec3e8ad9e46a036421b2139dc20f476a790f90a2dfb7" exitCode=60 Nov 02 23:15:23.483640 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:23.483534 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz" event={"ID":"6b3a4d68-657c-4da3-8fbf-891f0926b134","Type":"ContainerDied","Data":"7949375b59b23838ead5ec3e8ad9e46a036421b2139dc20f476a790f90a2dfb7"} Nov 02 23:15:23.483917 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:23.483896 5114 scope.go:122] "RemoveContainer" containerID="7949375b59b23838ead5ec3e8ad9e46a036421b2139dc20f476a790f90a2dfb7" Nov 02 23:15:23.484215 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:15:23.484190 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz_odh-ai-gateway-infra(6b3a4d68-657c-4da3-8fbf-891f0926b134)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz" podUID="6b3a4d68-657c-4da3-8fbf-891f0926b134" Nov 02 23:15:23.485561 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:23.485533 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7" event={"ID":"83a4dc67-3b2b-463f-a869-76c7d9b72195","Type":"ContainerStarted","Data":"236d2538c8e761b0915bee3d0069d4e4a6212c1ffcaf6fefe0cd7d5dbf16836b"} Nov 02 23:15:23.486853 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:23.486829 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26" event={"ID":"290360eb-55b0-42b3-9171-c74f9bb33bac","Type":"ContainerDied","Data":"d783a6aae4f05916b8ad4a3b10e7dc06ad4e2e19c234b13043dea02f9f4bfe9a"} Nov 02 23:15:23.486954 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:23.486855 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26" Nov 02 23:15:23.493940 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:23.493925 5114 scope.go:122] "RemoveContainer" containerID="dee0c0754de46a48fdfa71000209ffc4d18f088c176cdc85de8b44e6fa2008b7" Nov 02 23:15:23.507705 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:23.507673 5114 scope.go:122] "RemoveContainer" containerID="f6d4edf6c233499d7d17122be2c95ad556c40900048eb9c7226729256f4bf067" Nov 02 23:15:23.597832 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:23.597784 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5"] Nov 02 23:15:23.724767 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:23.724735 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26"] Nov 02 23:15:23.733028 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:23.732938 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894355-9gk26"] Nov 02 23:15:23.826912 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:23.826872 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz"] Nov 02 23:15:24.494458 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:24.494425 5114 generic.go:356] "Generic (PLEG): container finished" podID="83a4dc67-3b2b-463f-a869-76c7d9b72195" containerID="236d2538c8e761b0915bee3d0069d4e4a6212c1ffcaf6fefe0cd7d5dbf16836b" exitCode=60 Nov 02 23:15:24.494967 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:24.494507 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7" event={"ID":"83a4dc67-3b2b-463f-a869-76c7d9b72195","Type":"ContainerDied","Data":"236d2538c8e761b0915bee3d0069d4e4a6212c1ffcaf6fefe0cd7d5dbf16836b"} Nov 02 23:15:24.494967 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:24.494579 5114 scope.go:122] "RemoveContainer" containerID="2126f7a4eaf242ccd52f4abff716df9250b975301d772fd6ff1c8ae5b2a446c2" Nov 02 23:15:24.495086 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:24.494974 5114 scope.go:122] "RemoveContainer" containerID="236d2538c8e761b0915bee3d0069d4e4a6212c1ffcaf6fefe0cd7d5dbf16836b" Nov 02 23:15:24.495346 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:15:24.495319 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7_odh-ai-gateway-infra(83a4dc67-3b2b-463f-a869-76c7d9b72195)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7" podUID="83a4dc67-3b2b-463f-a869-76c7d9b72195" Nov 02 23:15:24.670881 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:24.670846 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz" Nov 02 23:15:24.712669 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:24.712641 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5" Nov 02 23:15:24.714887 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:24.714851 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b3a4d68-657c-4da3-8fbf-891f0926b134-kube-api-access-gddl9\" (UniqueName: \"kubernetes.io/projected/6b3a4d68-657c-4da3-8fbf-891f0926b134-kube-api-access-gddl9\") pod \"6b3a4d68-657c-4da3-8fbf-891f0926b134\" (UID: \"6b3a4d68-657c-4da3-8fbf-891f0926b134\") " Nov 02 23:15:24.715050 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:24.715012 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b3a4d68-657c-4da3-8fbf-891f0926b134-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3a4d68-657c-4da3-8fbf-891f0926b134-maas-api-service-ca\") pod \"6b3a4d68-657c-4da3-8fbf-891f0926b134\" (UID: \"6b3a4d68-657c-4da3-8fbf-891f0926b134\") " Nov 02 23:15:24.715345 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:24.715323 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3a4d68-657c-4da3-8fbf-891f0926b134-maas-api-service-ca" pod "6b3a4d68-657c-4da3-8fbf-891f0926b134" (UID: "6b3a4d68-657c-4da3-8fbf-891f0926b134"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:15:24.715506 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:24.715469 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3a4d68-657c-4da3-8fbf-891f0926b134-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:24.717469 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:24.717444 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3a4d68-657c-4da3-8fbf-891f0926b134-kube-api-access-gddl9" pod "6b3a4d68-657c-4da3-8fbf-891f0926b134" (UID: "6b3a4d68-657c-4da3-8fbf-891f0926b134"). InnerVolumeSpecName "kube-api-access-gddl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:15:24.794326 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:24.794293 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="290360eb-55b0-42b3-9171-c74f9bb33bac" path="/var/lib/kubelet/pods/290360eb-55b0-42b3-9171-c74f9bb33bac/volumes" Nov 02 23:15:24.816431 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:24.816400 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a1bf336-e967-4582-83b8-12584603fd63-kube-api-access-sqnvf\" (UniqueName: \"kubernetes.io/projected/3a1bf336-e967-4582-83b8-12584603fd63-kube-api-access-sqnvf\") pod \"3a1bf336-e967-4582-83b8-12584603fd63\" (UID: \"3a1bf336-e967-4582-83b8-12584603fd63\") " Nov 02 23:15:24.816614 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:24.816458 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a1bf336-e967-4582-83b8-12584603fd63-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1bf336-e967-4582-83b8-12584603fd63-maas-api-service-ca\") pod \"3a1bf336-e967-4582-83b8-12584603fd63\" (UID: \"3a1bf336-e967-4582-83b8-12584603fd63\") " Nov 02 23:15:24.816842 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:24.816823 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gddl9\" (UniqueName: \"kubernetes.io/projected/6b3a4d68-657c-4da3-8fbf-891f0926b134-kube-api-access-gddl9\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:24.816931 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:24.816832 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a1bf336-e967-4582-83b8-12584603fd63-maas-api-service-ca" pod "3a1bf336-e967-4582-83b8-12584603fd63" (UID: "3a1bf336-e967-4582-83b8-12584603fd63"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:15:24.818771 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:24.818747 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1bf336-e967-4582-83b8-12584603fd63-kube-api-access-sqnvf" pod "3a1bf336-e967-4582-83b8-12584603fd63" (UID: "3a1bf336-e967-4582-83b8-12584603fd63"). InnerVolumeSpecName "kube-api-access-sqnvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:15:24.856830 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:24.856781 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7"] Nov 02 23:15:24.918114 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:24.918071 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqnvf\" (UniqueName: \"kubernetes.io/projected/3a1bf336-e967-4582-83b8-12584603fd63-kube-api-access-sqnvf\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:24.918114 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:24.918112 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1bf336-e967-4582-83b8-12584603fd63-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:25.506759 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:25.506731 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5" Nov 02 23:15:25.506759 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:25.506736 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5" event={"ID":"3a1bf336-e967-4582-83b8-12584603fd63","Type":"ContainerDied","Data":"22f76864129297d89ce47a9a92ff655615d36889883c9be4c6b149f25caca89d"} Nov 02 23:15:25.507315 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:25.506817 5114 scope.go:122] "RemoveContainer" containerID="4d7b0f0fdafc596b5188226c815ed4187dd714b29460fbad6c716f2f3519c6a8" Nov 02 23:15:25.508355 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:25.508335 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz" Nov 02 23:15:25.508437 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:25.508337 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz" event={"ID":"6b3a4d68-657c-4da3-8fbf-891f0926b134","Type":"ContainerDied","Data":"e02b93076df2f9f66afeeb30d5fb581c794ec4871da3e623e0fc2f17e8d1b5b4"} Nov 02 23:15:25.517705 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:25.517683 5114 scope.go:122] "RemoveContainer" containerID="7949375b59b23838ead5ec3e8ad9e46a036421b2139dc20f476a790f90a2dfb7" Nov 02 23:15:25.685163 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:25.685127 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7" Nov 02 23:15:25.696514 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:25.696461 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz"] Nov 02 23:15:25.700606 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:25.700568 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894355-p7kfz"] Nov 02 23:15:25.729326 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:25.729289 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a4dc67-3b2b-463f-a869-76c7d9b72195-kube-api-access-w99w9\" (UniqueName: \"kubernetes.io/projected/83a4dc67-3b2b-463f-a869-76c7d9b72195-kube-api-access-w99w9\") pod \"83a4dc67-3b2b-463f-a869-76c7d9b72195\" (UID: \"83a4dc67-3b2b-463f-a869-76c7d9b72195\") " Nov 02 23:15:25.729435 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:25.729342 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83a4dc67-3b2b-463f-a869-76c7d9b72195-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a4dc67-3b2b-463f-a869-76c7d9b72195-maas-api-service-ca\") pod \"83a4dc67-3b2b-463f-a869-76c7d9b72195\" (UID: \"83a4dc67-3b2b-463f-a869-76c7d9b72195\") " Nov 02 23:15:25.729785 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:25.729762 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a4dc67-3b2b-463f-a869-76c7d9b72195-maas-api-service-ca" pod "83a4dc67-3b2b-463f-a869-76c7d9b72195" (UID: "83a4dc67-3b2b-463f-a869-76c7d9b72195"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:15:25.731718 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:25.731697 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a4dc67-3b2b-463f-a869-76c7d9b72195-kube-api-access-w99w9" pod "83a4dc67-3b2b-463f-a869-76c7d9b72195" (UID: "83a4dc67-3b2b-463f-a869-76c7d9b72195"). InnerVolumeSpecName "kube-api-access-w99w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:15:25.809377 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:25.809334 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5"] Nov 02 23:15:25.813176 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:25.813148 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8ab111-29894355-glpq5"] Nov 02 23:15:25.830641 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:25.830609 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w99w9\" (UniqueName: \"kubernetes.io/projected/83a4dc67-3b2b-463f-a869-76c7d9b72195-kube-api-access-w99w9\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:25.830641 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:25.830636 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a4dc67-3b2b-463f-a869-76c7d9b72195-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:26.519031 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:26.519002 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7" Nov 02 23:15:26.519563 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:26.519010 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7" event={"ID":"83a4dc67-3b2b-463f-a869-76c7d9b72195","Type":"ContainerDied","Data":"b7e3ae042ae95648c6266b2abfe305014a90b56413db76957b929667656a3800"} Nov 02 23:15:26.519563 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:26.519136 5114 scope.go:122] "RemoveContainer" containerID="236d2538c8e761b0915bee3d0069d4e4a6212c1ffcaf6fefe0cd7d5dbf16836b" Nov 02 23:15:26.686658 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:26.686621 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7"] Nov 02 23:15:26.735251 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:26.735216 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7a67875-29894355-b4tj7"] Nov 02 23:15:26.793746 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:26.793711 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a1bf336-e967-4582-83b8-12584603fd63" path="/var/lib/kubelet/pods/3a1bf336-e967-4582-83b8-12584603fd63/volumes" Nov 02 23:15:26.794345 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:26.794325 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b3a4d68-657c-4da3-8fbf-891f0926b134" path="/var/lib/kubelet/pods/6b3a4d68-657c-4da3-8fbf-891f0926b134/volumes" Nov 02 23:15:26.794896 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:26.794880 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83a4dc67-3b2b-463f-a869-76c7d9b72195" path="/var/lib/kubelet/pods/83a4dc67-3b2b-463f-a869-76c7d9b72195/volumes" Nov 02 23:15:37.587593 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:15:37.587542 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:15:53.902712 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:53.902662 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm"] Nov 02 23:15:53.903206 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:53.903034 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" podUID="ade938ea-2fe8-49c9-908f-78bbf9b02606" containerName="istio-proxy" containerID="cri-o://ce45ea75367581746591b8134e028b1b2a1c979f8b83c3a331cc98f6cc0a0058" gracePeriod=30 Nov 02 23:15:59.170016 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.169943 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:15:59.282321 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.282278 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-data\") pod \"ade938ea-2fe8-49c9-908f-78bbf9b02606\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " Nov 02 23:15:59.282568 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.282383 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-workload-socket\") pod \"ade938ea-2fe8-49c9-908f-78bbf9b02606\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " Nov 02 23:15:59.282568 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.282523 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-token\" (UniqueName: \"kubernetes.io/projected/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-token\") pod \"ade938ea-2fe8-49c9-908f-78bbf9b02606\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " Nov 02 23:15:59.282700 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.282577 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ade938ea-2fe8-49c9-908f-78bbf9b02606-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade938ea-2fe8-49c9-908f-78bbf9b02606-istiod-ca-cert\") pod \"ade938ea-2fe8-49c9-908f-78bbf9b02606\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " Nov 02 23:15:59.282700 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.282586 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-data" pod "ade938ea-2fe8-49c9-908f-78bbf9b02606" (UID: "ade938ea-2fe8-49c9-908f-78bbf9b02606"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:15:59.282700 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.282612 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-workload-certs\") pod \"ade938ea-2fe8-49c9-908f-78bbf9b02606\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " Nov 02 23:15:59.282700 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.282657 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-podinfo\") pod \"ade938ea-2fe8-49c9-908f-78bbf9b02606\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " Nov 02 23:15:59.282899 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.282716 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade938ea-2fe8-49c9-908f-78bbf9b02606-kube-api-access-49fm8\" (UniqueName: \"kubernetes.io/projected/ade938ea-2fe8-49c9-908f-78bbf9b02606-kube-api-access-49fm8\") pod \"ade938ea-2fe8-49c9-908f-78bbf9b02606\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " Nov 02 23:15:59.282899 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.282744 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-workload-socket" pod "ade938ea-2fe8-49c9-908f-78bbf9b02606" (UID: "ade938ea-2fe8-49c9-908f-78bbf9b02606"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:15:59.282899 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.282771 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-credential-socket\") pod \"ade938ea-2fe8-49c9-908f-78bbf9b02606\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " Nov 02 23:15:59.282899 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.282801 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-envoy\") pod \"ade938ea-2fe8-49c9-908f-78bbf9b02606\" (UID: \"ade938ea-2fe8-49c9-908f-78bbf9b02606\") " Nov 02 23:15:59.283102 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.283025 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-workload-certs" pod "ade938ea-2fe8-49c9-908f-78bbf9b02606" (UID: "ade938ea-2fe8-49c9-908f-78bbf9b02606"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:15:59.283155 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.283108 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade938ea-2fe8-49c9-908f-78bbf9b02606-istiod-ca-cert" pod "ade938ea-2fe8-49c9-908f-78bbf9b02606" (UID: "ade938ea-2fe8-49c9-908f-78bbf9b02606"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:15:59.283497 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.283375 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-credential-socket" pod "ade938ea-2fe8-49c9-908f-78bbf9b02606" (UID: "ade938ea-2fe8-49c9-908f-78bbf9b02606"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:15:59.283665 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.283646 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:59.283730 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.283668 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:59.283730 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.283683 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade938ea-2fe8-49c9-908f-78bbf9b02606-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:59.283730 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.283693 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:59.283730 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.283702 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:59.285297 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.285262 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-token" pod "ade938ea-2fe8-49c9-908f-78bbf9b02606" (UID: "ade938ea-2fe8-49c9-908f-78bbf9b02606"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:15:59.285417 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.285374 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-podinfo" pod "ade938ea-2fe8-49c9-908f-78bbf9b02606" (UID: "ade938ea-2fe8-49c9-908f-78bbf9b02606"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:15:59.285644 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.285617 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-envoy" pod "ade938ea-2fe8-49c9-908f-78bbf9b02606" (UID: "ade938ea-2fe8-49c9-908f-78bbf9b02606"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:15:59.285746 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.285653 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade938ea-2fe8-49c9-908f-78bbf9b02606-kube-api-access-49fm8" pod "ade938ea-2fe8-49c9-908f-78bbf9b02606" (UID: "ade938ea-2fe8-49c9-908f-78bbf9b02606"). InnerVolumeSpecName "kube-api-access-49fm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:15:59.385663 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.385614 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:59.385663 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.385662 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:59.385905 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.385676 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49fm8\" (UniqueName: \"kubernetes.io/projected/ade938ea-2fe8-49c9-908f-78bbf9b02606-kube-api-access-49fm8\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:59.385905 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.385689 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade938ea-2fe8-49c9-908f-78bbf9b02606-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:59.736534 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.736495 5114 generic.go:356] "Generic (PLEG): container finished" podID="ade938ea-2fe8-49c9-908f-78bbf9b02606" containerID="ce45ea75367581746591b8134e028b1b2a1c979f8b83c3a331cc98f6cc0a0058" exitCode=0 Nov 02 23:15:59.736847 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.736564 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" event={"ID":"ade938ea-2fe8-49c9-908f-78bbf9b02606","Type":"ContainerDied","Data":"ce45ea75367581746591b8134e028b1b2a1c979f8b83c3a331cc98f6cc0a0058"} Nov 02 23:15:59.736980 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.736893 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" event={"ID":"ade938ea-2fe8-49c9-908f-78bbf9b02606","Type":"ContainerDied","Data":"3c8cd7a11a51e8ffe64e6f7d2c6a6ff762128b30c4b483c15aaf7587dbf62501"} Nov 02 23:15:59.736980 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.736585 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm" Nov 02 23:15:59.736980 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.736933 5114 scope.go:122] "RemoveContainer" containerID="ce45ea75367581746591b8134e028b1b2a1c979f8b83c3a331cc98f6cc0a0058" Nov 02 23:15:59.748722 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.748698 5114 scope.go:122] "RemoveContainer" containerID="ce45ea75367581746591b8134e028b1b2a1c979f8b83c3a331cc98f6cc0a0058" Nov 02 23:15:59.749068 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:15:59.749047 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce45ea75367581746591b8134e028b1b2a1c979f8b83c3a331cc98f6cc0a0058\": container with ID starting with ce45ea75367581746591b8134e028b1b2a1c979f8b83c3a331cc98f6cc0a0058 not found: ID does not exist" containerID="ce45ea75367581746591b8134e028b1b2a1c979f8b83c3a331cc98f6cc0a0058" Nov 02 23:15:59.749128 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.749076 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce45ea75367581746591b8134e028b1b2a1c979f8b83c3a331cc98f6cc0a0058"} err="failed to get container status \"ce45ea75367581746591b8134e028b1b2a1c979f8b83c3a331cc98f6cc0a0058\": rpc error: code = NotFound desc = could not find container \"ce45ea75367581746591b8134e028b1b2a1c979f8b83c3a331cc98f6cc0a0058\": container with ID starting with ce45ea75367581746591b8134e028b1b2a1c979f8b83c3a331cc98f6cc0a0058 not found: ID does not exist" Nov 02 23:15:59.908679 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.908636 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm"] Nov 02 23:15:59.953792 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:15:59.953757 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-tv8fm"] Nov 02 23:16:00.797343 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:00.797302 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ade938ea-2fe8-49c9-908f-78bbf9b02606" path="/var/lib/kubelet/pods/ade938ea-2fe8-49c9-908f-78bbf9b02606/volumes" Nov 02 23:16:07.584868 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:16:07.584820 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:16:37.624445 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:16:37.624391 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:16:38.086117 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:38.086075 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b"] Nov 02 23:16:38.086361 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:38.086332 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" podUID="b45caa0e-af1c-45df-81af-f0ba37e27f16" containerName="istio-proxy" containerID="cri-o://55736e5fcb0c962a4a764c4cf6a3ca9d995aa5e22932b3f1a614f15aba13c870" gracePeriod=30 Nov 02 23:16:43.366815 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.366740 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:16:43.498337 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.498288 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b45caa0e-af1c-45df-81af-f0ba37e27f16-kube-api-access-ml55g\" (UniqueName: \"kubernetes.io/projected/b45caa0e-af1c-45df-81af-f0ba37e27f16-kube-api-access-ml55g\") pod \"b45caa0e-af1c-45df-81af-f0ba37e27f16\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " Nov 02 23:16:43.498537 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.498411 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-workload-certs\") pod \"b45caa0e-af1c-45df-81af-f0ba37e27f16\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " Nov 02 23:16:43.498537 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.498455 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-token\" (UniqueName: \"kubernetes.io/projected/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-token\") pod \"b45caa0e-af1c-45df-81af-f0ba37e27f16\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " Nov 02 23:16:43.498537 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.498505 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b45caa0e-af1c-45df-81af-f0ba37e27f16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45caa0e-af1c-45df-81af-f0ba37e27f16-istiod-ca-cert\") pod \"b45caa0e-af1c-45df-81af-f0ba37e27f16\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " Nov 02 23:16:43.498683 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.498566 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-envoy\") pod \"b45caa0e-af1c-45df-81af-f0ba37e27f16\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " Nov 02 23:16:43.498683 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.498606 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-workload-socket\") pod \"b45caa0e-af1c-45df-81af-f0ba37e27f16\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " Nov 02 23:16:43.498683 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.498645 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-credential-socket\") pod \"b45caa0e-af1c-45df-81af-f0ba37e27f16\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " Nov 02 23:16:43.498835 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.498719 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-podinfo\") pod \"b45caa0e-af1c-45df-81af-f0ba37e27f16\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " Nov 02 23:16:43.498835 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.498829 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-data\") pod \"b45caa0e-af1c-45df-81af-f0ba37e27f16\" (UID: \"b45caa0e-af1c-45df-81af-f0ba37e27f16\") " Nov 02 23:16:43.498931 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.498854 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-workload-certs" pod "b45caa0e-af1c-45df-81af-f0ba37e27f16" (UID: "b45caa0e-af1c-45df-81af-f0ba37e27f16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:43.499006 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.498974 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-workload-socket" pod "b45caa0e-af1c-45df-81af-f0ba37e27f16" (UID: "b45caa0e-af1c-45df-81af-f0ba37e27f16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:43.499315 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.499257 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-credential-socket" pod "b45caa0e-af1c-45df-81af-f0ba37e27f16" (UID: "b45caa0e-af1c-45df-81af-f0ba37e27f16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:43.499315 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.499278 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-data" pod "b45caa0e-af1c-45df-81af-f0ba37e27f16" (UID: "b45caa0e-af1c-45df-81af-f0ba37e27f16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:43.499631 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.499611 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45caa0e-af1c-45df-81af-f0ba37e27f16-istiod-ca-cert" pod "b45caa0e-af1c-45df-81af-f0ba37e27f16" (UID: "b45caa0e-af1c-45df-81af-f0ba37e27f16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:16:43.499704 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.499624 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.499704 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.499649 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.499704 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.499666 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.499704 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.499681 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.501060 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.501022 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-token" pod "b45caa0e-af1c-45df-81af-f0ba37e27f16" (UID: "b45caa0e-af1c-45df-81af-f0ba37e27f16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:43.501640 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.501616 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-podinfo" pod "b45caa0e-af1c-45df-81af-f0ba37e27f16" (UID: "b45caa0e-af1c-45df-81af-f0ba37e27f16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:16:43.501725 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.501614 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-envoy" pod "b45caa0e-af1c-45df-81af-f0ba37e27f16" (UID: "b45caa0e-af1c-45df-81af-f0ba37e27f16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:43.502125 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.502100 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45caa0e-af1c-45df-81af-f0ba37e27f16-kube-api-access-ml55g" pod "b45caa0e-af1c-45df-81af-f0ba37e27f16" (UID: "b45caa0e-af1c-45df-81af-f0ba37e27f16"). InnerVolumeSpecName "kube-api-access-ml55g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:43.600852 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.600821 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml55g\" (UniqueName: \"kubernetes.io/projected/b45caa0e-af1c-45df-81af-f0ba37e27f16-kube-api-access-ml55g\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.600852 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.600849 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.600852 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.600859 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45caa0e-af1c-45df-81af-f0ba37e27f16-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.601208 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.600870 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.601208 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:43.600878 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45caa0e-af1c-45df-81af-f0ba37e27f16-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:44.024677 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:44.024637 5114 generic.go:356] "Generic (PLEG): container finished" podID="b45caa0e-af1c-45df-81af-f0ba37e27f16" containerID="55736e5fcb0c962a4a764c4cf6a3ca9d995aa5e22932b3f1a614f15aba13c870" exitCode=0 Nov 02 23:16:44.024677 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:44.024687 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" Nov 02 23:16:44.024945 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:44.024711 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" event={"ID":"b45caa0e-af1c-45df-81af-f0ba37e27f16","Type":"ContainerDied","Data":"55736e5fcb0c962a4a764c4cf6a3ca9d995aa5e22932b3f1a614f15aba13c870"} Nov 02 23:16:44.024945 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:44.024767 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b" event={"ID":"b45caa0e-af1c-45df-81af-f0ba37e27f16","Type":"ContainerDied","Data":"ecf1d7cfd94a4cd673ca4671daa677752fa7f7598edffff9a0b502c755cc4927"} Nov 02 23:16:44.024945 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:44.024817 5114 scope.go:122] "RemoveContainer" containerID="55736e5fcb0c962a4a764c4cf6a3ca9d995aa5e22932b3f1a614f15aba13c870" Nov 02 23:16:44.036740 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:44.036713 5114 scope.go:122] "RemoveContainer" containerID="55736e5fcb0c962a4a764c4cf6a3ca9d995aa5e22932b3f1a614f15aba13c870" Nov 02 23:16:44.037268 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:16:44.037239 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55736e5fcb0c962a4a764c4cf6a3ca9d995aa5e22932b3f1a614f15aba13c870\": container with ID starting with 55736e5fcb0c962a4a764c4cf6a3ca9d995aa5e22932b3f1a614f15aba13c870 not found: ID does not exist" containerID="55736e5fcb0c962a4a764c4cf6a3ca9d995aa5e22932b3f1a614f15aba13c870" Nov 02 23:16:44.037385 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:44.037274 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55736e5fcb0c962a4a764c4cf6a3ca9d995aa5e22932b3f1a614f15aba13c870"} err="failed to get container status \"55736e5fcb0c962a4a764c4cf6a3ca9d995aa5e22932b3f1a614f15aba13c870\": rpc error: code = NotFound desc = could not find container \"55736e5fcb0c962a4a764c4cf6a3ca9d995aa5e22932b3f1a614f15aba13c870\": container with ID starting with 55736e5fcb0c962a4a764c4cf6a3ca9d995aa5e22932b3f1a614f15aba13c870 not found: ID does not exist" Nov 02 23:16:44.189888 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:44.189844 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b"] Nov 02 23:16:44.194032 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:44.194001 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-xwc2b"] Nov 02 23:16:44.793931 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:16:44.793896 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b45caa0e-af1c-45df-81af-f0ba37e27f16" path="/var/lib/kubelet/pods/b45caa0e-af1c-45df-81af-f0ba37e27f16/volumes" Nov 02 23:17:07.585287 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:17:07.585246 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:17:37.584251 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:17:37.584213 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:18:07.601072 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:18:07.601030 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:18:22.044068 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:22.044025 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9"] Nov 02 23:18:37.066852 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.066793 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" podUID="d8f5d813-1cf7-47f0-b45e-230a313e7a1f" containerName="main" containerID="cri-o://0e7ef3e88564cc4b1765203fe4d82c0ba4109cdfa449f470532717a9806ed5fc" gracePeriod=45 Nov 02 23:18:37.241123 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.240778 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" podUID="d8f5d813-1cf7-47f0-b45e-230a313e7a1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": dial tcp 10.134.0.60:8000: connect: connection refused" Nov 02 23:18:37.350177 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.350133 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:18:37.503801 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.503765 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-tmp-dir\") pod \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\" (UID: \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\") " Nov 02 23:18:37.504021 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.503844 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-dshm\") pod \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\" (UID: \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\") " Nov 02 23:18:37.504021 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.503878 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-home\" (UniqueName: \"kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-home\") pod \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\" (UID: \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\") " Nov 02 23:18:37.504021 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.503917 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-kube-api-access-pvst6\" (UniqueName: \"kubernetes.io/projected/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-kube-api-access-pvst6\") pod \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\" (UID: \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\") " Nov 02 23:18:37.504021 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.503987 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-tls-certs\") pod \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\" (UID: \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\") " Nov 02 23:18:37.504021 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.504007 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-tmp-dir" pod "d8f5d813-1cf7-47f0-b45e-230a313e7a1f" (UID: "d8f5d813-1cf7-47f0-b45e-230a313e7a1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:37.504310 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.504043 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-model-cache\") pod \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\" (UID: \"d8f5d813-1cf7-47f0-b45e-230a313e7a1f\") " Nov 02 23:18:37.504310 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.504169 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-home" pod "d8f5d813-1cf7-47f0-b45e-230a313e7a1f" (UID: "d8f5d813-1cf7-47f0-b45e-230a313e7a1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:37.504592 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.504566 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:37.504765 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.504744 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:37.504765 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.504596 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-model-cache" pod "d8f5d813-1cf7-47f0-b45e-230a313e7a1f" (UID: "d8f5d813-1cf7-47f0-b45e-230a313e7a1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:37.506782 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.506753 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-tls-certs" pod "d8f5d813-1cf7-47f0-b45e-230a313e7a1f" (UID: "d8f5d813-1cf7-47f0-b45e-230a313e7a1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:18:37.506965 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.506939 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-kube-api-access-pvst6" pod "d8f5d813-1cf7-47f0-b45e-230a313e7a1f" (UID: "d8f5d813-1cf7-47f0-b45e-230a313e7a1f"). InnerVolumeSpecName "kube-api-access-pvst6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:37.507246 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.507217 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-dshm" pod "d8f5d813-1cf7-47f0-b45e-230a313e7a1f" (UID: "d8f5d813-1cf7-47f0-b45e-230a313e7a1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:37.588065 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:18:37.587974 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:18:37.606139 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.606102 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:37.606139 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.606133 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvst6\" (UniqueName: \"kubernetes.io/projected/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-kube-api-access-pvst6\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:37.606139 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.606147 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:37.606383 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.606158 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8f5d813-1cf7-47f0-b45e-230a313e7a1f-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:37.752563 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.752520 5114 generic.go:356] "Generic (PLEG): container finished" podID="d8f5d813-1cf7-47f0-b45e-230a313e7a1f" containerID="0e7ef3e88564cc4b1765203fe4d82c0ba4109cdfa449f470532717a9806ed5fc" exitCode=0 Nov 02 23:18:37.752563 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.752567 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" Nov 02 23:18:37.752785 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.752582 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" event={"ID":"d8f5d813-1cf7-47f0-b45e-230a313e7a1f","Type":"ContainerDied","Data":"0e7ef3e88564cc4b1765203fe4d82c0ba4109cdfa449f470532717a9806ed5fc"} Nov 02 23:18:37.752785 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.752635 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9" event={"ID":"d8f5d813-1cf7-47f0-b45e-230a313e7a1f","Type":"ContainerDied","Data":"e6bf2877bc0f76c6c532c595ead6edd4ee7fc0049c2a2d9bed2ec2c383e06e3c"} Nov 02 23:18:37.752785 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.752656 5114 scope.go:122] "RemoveContainer" containerID="0e7ef3e88564cc4b1765203fe4d82c0ba4109cdfa449f470532717a9806ed5fc" Nov 02 23:18:37.762847 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.762829 5114 scope.go:122] "RemoveContainer" containerID="0e7ef3e88564cc4b1765203fe4d82c0ba4109cdfa449f470532717a9806ed5fc" Nov 02 23:18:37.763155 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:18:37.763135 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e7ef3e88564cc4b1765203fe4d82c0ba4109cdfa449f470532717a9806ed5fc\": container with ID starting with 0e7ef3e88564cc4b1765203fe4d82c0ba4109cdfa449f470532717a9806ed5fc not found: ID does not exist" containerID="0e7ef3e88564cc4b1765203fe4d82c0ba4109cdfa449f470532717a9806ed5fc" Nov 02 23:18:37.763208 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.763166 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e7ef3e88564cc4b1765203fe4d82c0ba4109cdfa449f470532717a9806ed5fc"} err="failed to get container status \"0e7ef3e88564cc4b1765203fe4d82c0ba4109cdfa449f470532717a9806ed5fc\": rpc error: code = NotFound desc = could not find container \"0e7ef3e88564cc4b1765203fe4d82c0ba4109cdfa449f470532717a9806ed5fc\": container with ID starting with 0e7ef3e88564cc4b1765203fe4d82c0ba4109cdfa449f470532717a9806ed5fc not found: ID does not exist" Nov 02 23:18:37.917181 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.917092 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9"] Nov 02 23:18:37.966520 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:37.966460 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-a7a67875/e2e-lifecycle-model-a7a67875-kserve-6494c76597-ncmn9"] Nov 02 23:18:38.794129 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:18:38.794080 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8f5d813-1cf7-47f0-b45e-230a313e7a1f" path="/var/lib/kubelet/pods/d8f5d813-1cf7-47f0-b45e-230a313e7a1f/volumes" Nov 02 23:19:07.584465 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:19:07.584420 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:19:37.601986 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:19:37.601925 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:19:47.836674 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.836635 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx"] Nov 02 23:19:47.837177 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837146 5114 cpu_manager.go:414] "containerMap: removing container" podUID="b45caa0e-af1c-45df-81af-f0ba37e27f16" containerName="istio-proxy" Nov 02 23:19:47.837177 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837159 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b45caa0e-af1c-45df-81af-f0ba37e27f16" containerName="istio-proxy" Nov 02 23:19:47.837177 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837178 5114 cpu_manager.go:414] "containerMap: removing container" podUID="ade938ea-2fe8-49c9-908f-78bbf9b02606" containerName="istio-proxy" Nov 02 23:19:47.837284 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837183 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade938ea-2fe8-49c9-908f-78bbf9b02606" containerName="istio-proxy" Nov 02 23:19:47.837284 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837190 5114 cpu_manager.go:414] "containerMap: removing container" podUID="3a1bf336-e967-4582-83b8-12584603fd63" containerName="cleanup" Nov 02 23:19:47.837284 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837195 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1bf336-e967-4582-83b8-12584603fd63" containerName="cleanup" Nov 02 23:19:47.837284 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837201 5114 cpu_manager.go:414] "containerMap: removing container" podUID="6b3a4d68-657c-4da3-8fbf-891f0926b134" containerName="cleanup" Nov 02 23:19:47.837284 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837205 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3a4d68-657c-4da3-8fbf-891f0926b134" containerName="cleanup" Nov 02 23:19:47.837284 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837214 5114 cpu_manager.go:414] "containerMap: removing container" podUID="6b3a4d68-657c-4da3-8fbf-891f0926b134" containerName="cleanup" Nov 02 23:19:47.837284 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837218 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3a4d68-657c-4da3-8fbf-891f0926b134" containerName="cleanup" Nov 02 23:19:47.837284 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837224 5114 cpu_manager.go:414] "containerMap: removing container" podUID="83a4dc67-3b2b-463f-a869-76c7d9b72195" containerName="cleanup" Nov 02 23:19:47.837284 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837229 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a4dc67-3b2b-463f-a869-76c7d9b72195" containerName="cleanup" Nov 02 23:19:47.837284 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837236 5114 cpu_manager.go:414] "containerMap: removing container" podUID="d8f5d813-1cf7-47f0-b45e-230a313e7a1f" containerName="main" Nov 02 23:19:47.837284 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837240 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8f5d813-1cf7-47f0-b45e-230a313e7a1f" containerName="main" Nov 02 23:19:47.837284 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837246 5114 cpu_manager.go:414] "containerMap: removing container" podUID="033ac52a-46b6-4b42-b9c0-b5f31a6b1675" containerName="revoke-keys" Nov 02 23:19:47.837284 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837251 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="033ac52a-46b6-4b42-b9c0-b5f31a6b1675" containerName="revoke-keys" Nov 02 23:19:47.837284 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837264 5114 cpu_manager.go:414] "containerMap: removing container" podUID="3a1bf336-e967-4582-83b8-12584603fd63" containerName="cleanup" Nov 02 23:19:47.837284 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837268 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1bf336-e967-4582-83b8-12584603fd63" containerName="cleanup" Nov 02 23:19:47.837284 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837273 5114 cpu_manager.go:414] "containerMap: removing container" podUID="83a4dc67-3b2b-463f-a869-76c7d9b72195" containerName="cleanup" Nov 02 23:19:47.837284 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837277 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a4dc67-3b2b-463f-a869-76c7d9b72195" containerName="cleanup" Nov 02 23:19:47.837284 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837285 5114 cpu_manager.go:414] "containerMap: removing container" podUID="83a4dc67-3b2b-463f-a869-76c7d9b72195" containerName="cleanup" Nov 02 23:19:47.837284 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837290 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a4dc67-3b2b-463f-a869-76c7d9b72195" containerName="cleanup" Nov 02 23:19:47.837861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837299 5114 cpu_manager.go:414] "containerMap: removing container" podUID="290360eb-55b0-42b3-9171-c74f9bb33bac" containerName="cleanup" Nov 02 23:19:47.837861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837303 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="290360eb-55b0-42b3-9171-c74f9bb33bac" containerName="cleanup" Nov 02 23:19:47.837861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837314 5114 cpu_manager.go:414] "containerMap: removing container" podUID="290360eb-55b0-42b3-9171-c74f9bb33bac" containerName="cleanup" Nov 02 23:19:47.837861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837318 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="290360eb-55b0-42b3-9171-c74f9bb33bac" containerName="cleanup" Nov 02 23:19:47.837861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837324 5114 cpu_manager.go:414] "containerMap: removing container" podUID="290360eb-55b0-42b3-9171-c74f9bb33bac" containerName="cleanup" Nov 02 23:19:47.837861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837328 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="290360eb-55b0-42b3-9171-c74f9bb33bac" containerName="cleanup" Nov 02 23:19:47.837861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837405 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="290360eb-55b0-42b3-9171-c74f9bb33bac" containerName="cleanup" Nov 02 23:19:47.837861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837411 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1bf336-e967-4582-83b8-12584603fd63" containerName="cleanup" Nov 02 23:19:47.837861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837416 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="290360eb-55b0-42b3-9171-c74f9bb33bac" containerName="cleanup" Nov 02 23:19:47.837861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837423 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3a4d68-657c-4da3-8fbf-891f0926b134" containerName="cleanup" Nov 02 23:19:47.837861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837428 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="290360eb-55b0-42b3-9171-c74f9bb33bac" containerName="cleanup" Nov 02 23:19:47.837861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837435 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="b45caa0e-af1c-45df-81af-f0ba37e27f16" containerName="istio-proxy" Nov 02 23:19:47.837861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837443 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade938ea-2fe8-49c9-908f-78bbf9b02606" containerName="istio-proxy" Nov 02 23:19:47.837861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837448 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1bf336-e967-4582-83b8-12584603fd63" containerName="cleanup" Nov 02 23:19:47.837861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837452 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3a4d68-657c-4da3-8fbf-891f0926b134" containerName="cleanup" Nov 02 23:19:47.837861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837457 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3a4d68-657c-4da3-8fbf-891f0926b134" containerName="cleanup" Nov 02 23:19:47.837861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837462 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a4dc67-3b2b-463f-a869-76c7d9b72195" containerName="cleanup" Nov 02 23:19:47.837861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837467 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8f5d813-1cf7-47f0-b45e-230a313e7a1f" containerName="main" Nov 02 23:19:47.837861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837472 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="033ac52a-46b6-4b42-b9c0-b5f31a6b1675" containerName="revoke-keys" Nov 02 23:19:47.837861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837492 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a4dc67-3b2b-463f-a869-76c7d9b72195" containerName="cleanup" Nov 02 23:19:47.837861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.837497 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a4dc67-3b2b-463f-a869-76c7d9b72195" containerName="cleanup" Nov 02 23:19:47.839788 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.839773 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx" Nov 02 23:19:47.894901 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.894764 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:19:47.894901 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.894869 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:19:47.917146 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.917097 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa02eae2-80f1-4a52-a122-1f2451ea4ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx\" (UID: \"fa02eae2-80f1-4a52-a122-1f2451ea4ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx" Nov 02 23:19:47.917357 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.917238 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8gv5\" (UniqueName: \"kubernetes.io/projected/fa02eae2-80f1-4a52-a122-1f2451ea4ebd-kube-api-access-d8gv5\") pod \"maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx\" (UID: \"fa02eae2-80f1-4a52-a122-1f2451ea4ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx" Nov 02 23:19:47.917357 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.917284 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa02eae2-80f1-4a52-a122-1f2451ea4ebd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx\" (UID: \"fa02eae2-80f1-4a52-a122-1f2451ea4ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx" Nov 02 23:19:47.925785 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:47.925736 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx"] Nov 02 23:19:48.018938 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:48.018900 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa02eae2-80f1-4a52-a122-1f2451ea4ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx\" (UID: \"fa02eae2-80f1-4a52-a122-1f2451ea4ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx" Nov 02 23:19:48.019138 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:48.018957 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8gv5\" (UniqueName: \"kubernetes.io/projected/fa02eae2-80f1-4a52-a122-1f2451ea4ebd-kube-api-access-d8gv5\") pod \"maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx\" (UID: \"fa02eae2-80f1-4a52-a122-1f2451ea4ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx" Nov 02 23:19:48.019138 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:48.018980 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa02eae2-80f1-4a52-a122-1f2451ea4ebd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx\" (UID: \"fa02eae2-80f1-4a52-a122-1f2451ea4ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx" Nov 02 23:19:48.019802 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:48.019776 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa02eae2-80f1-4a52-a122-1f2451ea4ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx\" (UID: \"fa02eae2-80f1-4a52-a122-1f2451ea4ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx" Nov 02 23:19:48.021511 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:48.021473 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa02eae2-80f1-4a52-a122-1f2451ea4ebd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx\" (UID: \"fa02eae2-80f1-4a52-a122-1f2451ea4ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx" Nov 02 23:19:48.079664 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:48.079622 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8gv5\" (UniqueName: \"kubernetes.io/projected/fa02eae2-80f1-4a52-a122-1f2451ea4ebd-kube-api-access-d8gv5\") pod \"maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx\" (UID: \"fa02eae2-80f1-4a52-a122-1f2451ea4ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx" Nov 02 23:19:48.152395 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:48.152297 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx" Nov 02 23:19:48.537971 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:48.537936 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx"] Nov 02 23:19:48.539430 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:19:48.539396 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa02eae2_80f1_4a52_a122_1f2451ea4ebd.slice/crio-a76fce6a3c95364aa544aa2df2f1659e5b1514b995e05cab353db6a81adae000 WatchSource:0}: Error finding container a76fce6a3c95364aa544aa2df2f1659e5b1514b995e05cab353db6a81adae000: Status 404 returned error can't find the container with id a76fce6a3c95364aa544aa2df2f1659e5b1514b995e05cab353db6a81adae000 Nov 02 23:19:49.266682 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:49.266596 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx" event={"ID":"fa02eae2-80f1-4a52-a122-1f2451ea4ebd","Type":"ContainerStarted","Data":"7727144108d098a896ebf55ad26a177f6dbdc314473887ce48e3c3584f019c4b"} Nov 02 23:19:49.266682 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:49.266648 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx" event={"ID":"fa02eae2-80f1-4a52-a122-1f2451ea4ebd","Type":"ContainerStarted","Data":"a76fce6a3c95364aa544aa2df2f1659e5b1514b995e05cab353db6a81adae000"} Nov 02 23:19:49.382973 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:49.382917 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx" podStartSLOduration=2.382900783 podStartE2EDuration="2.382900783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:19:49.380336685 +0000 UTC m=+1864.733370060" watchObservedRunningTime="2026-11-02 23:19:49.382900783 +0000 UTC m=+1864.735934201" Nov 02 23:19:50.274171 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:50.274134 5114 generic.go:356] "Generic (PLEG): container finished" podID="fa02eae2-80f1-4a52-a122-1f2451ea4ebd" containerID="7727144108d098a896ebf55ad26a177f6dbdc314473887ce48e3c3584f019c4b" exitCode=0 Nov 02 23:19:50.274796 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:50.274215 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx" event={"ID":"fa02eae2-80f1-4a52-a122-1f2451ea4ebd","Type":"ContainerDied","Data":"7727144108d098a896ebf55ad26a177f6dbdc314473887ce48e3c3584f019c4b"} Nov 02 23:19:51.446799 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:51.446765 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx" Nov 02 23:19:51.573344 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:51.573301 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa02eae2-80f1-4a52-a122-1f2451ea4ebd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa02eae2-80f1-4a52-a122-1f2451ea4ebd-maas-api-xks-ca\") pod \"fa02eae2-80f1-4a52-a122-1f2451ea4ebd\" (UID: \"fa02eae2-80f1-4a52-a122-1f2451ea4ebd\") " Nov 02 23:19:51.573540 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:51.573475 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa02eae2-80f1-4a52-a122-1f2451ea4ebd-kube-api-access-d8gv5\" (UniqueName: \"kubernetes.io/projected/fa02eae2-80f1-4a52-a122-1f2451ea4ebd-kube-api-access-d8gv5\") pod \"fa02eae2-80f1-4a52-a122-1f2451ea4ebd\" (UID: \"fa02eae2-80f1-4a52-a122-1f2451ea4ebd\") " Nov 02 23:19:51.573633 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:51.573617 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa02eae2-80f1-4a52-a122-1f2451ea4ebd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa02eae2-80f1-4a52-a122-1f2451ea4ebd-maas-api-openshift-ca\") pod \"fa02eae2-80f1-4a52-a122-1f2451ea4ebd\" (UID: \"fa02eae2-80f1-4a52-a122-1f2451ea4ebd\") " Nov 02 23:19:51.574053 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:51.574021 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa02eae2-80f1-4a52-a122-1f2451ea4ebd-maas-api-openshift-ca" pod "fa02eae2-80f1-4a52-a122-1f2451ea4ebd" (UID: "fa02eae2-80f1-4a52-a122-1f2451ea4ebd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:19:51.574328 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:51.574304 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa02eae2-80f1-4a52-a122-1f2451ea4ebd-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:51.576371 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:51.576338 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa02eae2-80f1-4a52-a122-1f2451ea4ebd-kube-api-access-d8gv5" pod "fa02eae2-80f1-4a52-a122-1f2451ea4ebd" (UID: "fa02eae2-80f1-4a52-a122-1f2451ea4ebd"). InnerVolumeSpecName "kube-api-access-d8gv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:51.579627 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:51.579594 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa02eae2-80f1-4a52-a122-1f2451ea4ebd-maas-api-xks-ca" pod "fa02eae2-80f1-4a52-a122-1f2451ea4ebd" (UID: "fa02eae2-80f1-4a52-a122-1f2451ea4ebd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:19:51.676553 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:51.676511 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8gv5\" (UniqueName: \"kubernetes.io/projected/fa02eae2-80f1-4a52-a122-1f2451ea4ebd-kube-api-access-d8gv5\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:51.676553 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:51.676548 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa02eae2-80f1-4a52-a122-1f2451ea4ebd-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:52.289936 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:52.289907 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx" Nov 02 23:19:52.290132 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:52.289934 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx" event={"ID":"fa02eae2-80f1-4a52-a122-1f2451ea4ebd","Type":"ContainerDied","Data":"a76fce6a3c95364aa544aa2df2f1659e5b1514b995e05cab353db6a81adae000"} Nov 02 23:19:52.290132 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:52.289984 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a76fce6a3c95364aa544aa2df2f1659e5b1514b995e05cab353db6a81adae000" Nov 02 23:19:58.988536 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:58.988467 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d"] Nov 02 23:19:58.989050 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:58.988746 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" podUID="4df160f4-4ae0-4d2f-8464-5a7e0e792434" containerName="istio-proxy" containerID="cri-o://9d418505c24e6abab1e4b7a6c5ff84eb0374134b67844c3b8fc60b8833f96465" gracePeriod=30 Nov 02 23:19:59.040098 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:59.040050 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4"] Nov 02 23:19:59.040393 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:59.040358 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" podUID="95e2263e-c273-4505-baf4-65edea02ecd1" containerName="istio-proxy" containerID="cri-o://3cfd00ed5b081a10f19b01b68046d16ccc9d45085cdcca6f78ff81bea5f09684" gracePeriod=30 Nov 02 23:19:59.580430 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:59.580391 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:19:59.580638 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:59.580465 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" podUID="95e2263e-c273-4505-baf4-65edea02ecd1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:19:59.615615 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:59.615583 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:19:59.615781 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:19:59.615634 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" podUID="4df160f4-4ae0-4d2f-8464-5a7e0e792434" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:20:02.368098 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:02.368063 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx"] Nov 02 23:20:02.411383 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:02.411344 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-jmbsx"] Nov 02 23:20:02.811940 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:02.811898 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa02eae2-80f1-4a52-a122-1f2451ea4ebd" path="/var/lib/kubelet/pods/fa02eae2-80f1-4a52-a122-1f2451ea4ebd/volumes" Nov 02 23:20:04.387113 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.387073 5114 generic.go:356] "Generic (PLEG): container finished" podID="4df160f4-4ae0-4d2f-8464-5a7e0e792434" containerID="9d418505c24e6abab1e4b7a6c5ff84eb0374134b67844c3b8fc60b8833f96465" exitCode=0 Nov 02 23:20:04.387647 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.387246 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" event={"ID":"4df160f4-4ae0-4d2f-8464-5a7e0e792434","Type":"ContainerDied","Data":"9d418505c24e6abab1e4b7a6c5ff84eb0374134b67844c3b8fc60b8833f96465"} Nov 02 23:20:04.387647 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.387300 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" event={"ID":"4df160f4-4ae0-4d2f-8464-5a7e0e792434","Type":"ContainerDied","Data":"df2559d4fa7cf8b372d8e59c1c0df6393fcd989b3b7c9216e7d8a54e83a2ba16"} Nov 02 23:20:04.387647 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.387322 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df2559d4fa7cf8b372d8e59c1c0df6393fcd989b3b7c9216e7d8a54e83a2ba16" Nov 02 23:20:04.396868 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.396836 5114 generic.go:356] "Generic (PLEG): container finished" podID="95e2263e-c273-4505-baf4-65edea02ecd1" containerID="3cfd00ed5b081a10f19b01b68046d16ccc9d45085cdcca6f78ff81bea5f09684" exitCode=0 Nov 02 23:20:04.397058 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.396952 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" event={"ID":"95e2263e-c273-4505-baf4-65edea02ecd1","Type":"ContainerDied","Data":"3cfd00ed5b081a10f19b01b68046d16ccc9d45085cdcca6f78ff81bea5f09684"} Nov 02 23:20:04.421372 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.421340 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:20:04.474003 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.473974 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:20:04.499861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.499808 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-workload-certs\") pod \"95e2263e-c273-4505-baf4-65edea02ecd1\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " Nov 02 23:20:04.499861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.499871 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-envoy\") pod \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " Nov 02 23:20:04.500131 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.499909 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-workload-socket\") pod \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " Nov 02 23:20:04.500131 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.499981 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-data\") pod \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " Nov 02 23:20:04.500131 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.500031 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-credential-socket\") pod \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " Nov 02 23:20:04.500131 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.500071 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95e2263e-c273-4505-baf4-65edea02ecd1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95e2263e-c273-4505-baf4-65edea02ecd1-istio-podinfo\") pod \"95e2263e-c273-4505-baf4-65edea02ecd1\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " Nov 02 23:20:04.500131 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.500102 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istiod-ca-cert\") pod \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " Nov 02 23:20:04.500131 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.500115 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-workload-socket" pod "4df160f4-4ae0-4d2f-8464-5a7e0e792434" (UID: "4df160f4-4ae0-4d2f-8464-5a7e0e792434"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:04.500428 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.500151 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-workload-certs" pod "95e2263e-c273-4505-baf4-65edea02ecd1" (UID: "95e2263e-c273-4505-baf4-65edea02ecd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:04.500428 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.500193 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-istio-data\") pod \"95e2263e-c273-4505-baf4-65edea02ecd1\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " Nov 02 23:20:04.500428 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.500222 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95e2263e-c273-4505-baf4-65edea02ecd1-kube-api-access-n5fcr\" (UniqueName: \"kubernetes.io/projected/95e2263e-c273-4505-baf4-65edea02ecd1-kube-api-access-n5fcr\") pod \"95e2263e-c273-4505-baf4-65edea02ecd1\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " Nov 02 23:20:04.500428 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.500258 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-workload-socket\") pod \"95e2263e-c273-4505-baf4-65edea02ecd1\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " Nov 02 23:20:04.500428 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.500293 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-istio-envoy\") pod \"95e2263e-c273-4505-baf4-65edea02ecd1\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " Nov 02 23:20:04.500428 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.500322 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-token\" (UniqueName: \"kubernetes.io/projected/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-token\") pod \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " Nov 02 23:20:04.500428 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.500352 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-credential-socket" pod "4df160f4-4ae0-4d2f-8464-5a7e0e792434" (UID: "4df160f4-4ae0-4d2f-8464-5a7e0e792434"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:04.500428 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.500423 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-workload-certs\") pod \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " Nov 02 23:20:04.500855 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.500469 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-credential-socket\") pod \"95e2263e-c273-4505-baf4-65edea02ecd1\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " Nov 02 23:20:04.500855 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.500551 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95e2263e-c273-4505-baf4-65edea02ecd1-istio-token\" (UniqueName: \"kubernetes.io/projected/95e2263e-c273-4505-baf4-65edea02ecd1-istio-token\") pod \"95e2263e-c273-4505-baf4-65edea02ecd1\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " Nov 02 23:20:04.500855 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.500604 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4df160f4-4ae0-4d2f-8464-5a7e0e792434-kube-api-access-fmjz6\" (UniqueName: \"kubernetes.io/projected/4df160f4-4ae0-4d2f-8464-5a7e0e792434-kube-api-access-fmjz6\") pod \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " Nov 02 23:20:04.500855 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.500646 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-podinfo\") pod \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\" (UID: \"4df160f4-4ae0-4d2f-8464-5a7e0e792434\") " Nov 02 23:20:04.500855 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.500700 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95e2263e-c273-4505-baf4-65edea02ecd1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95e2263e-c273-4505-baf4-65edea02ecd1-istiod-ca-cert\") pod \"95e2263e-c273-4505-baf4-65edea02ecd1\" (UID: \"95e2263e-c273-4505-baf4-65edea02ecd1\") " Nov 02 23:20:04.501586 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.501527 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:04.501739 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.501589 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:04.501739 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.501608 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:04.503120 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.500509 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-istio-data" pod "95e2263e-c273-4505-baf4-65edea02ecd1" (UID: "95e2263e-c273-4505-baf4-65edea02ecd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:04.503120 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.500583 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-workload-socket" pod "95e2263e-c273-4505-baf4-65edea02ecd1" (UID: "95e2263e-c273-4505-baf4-65edea02ecd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:04.503307 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.500613 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istiod-ca-cert" pod "4df160f4-4ae0-4d2f-8464-5a7e0e792434" (UID: "4df160f4-4ae0-4d2f-8464-5a7e0e792434"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:20:04.503307 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.500696 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-data" pod "4df160f4-4ae0-4d2f-8464-5a7e0e792434" (UID: "4df160f4-4ae0-4d2f-8464-5a7e0e792434"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:04.503307 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.500812 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-workload-certs" pod "4df160f4-4ae0-4d2f-8464-5a7e0e792434" (UID: "4df160f4-4ae0-4d2f-8464-5a7e0e792434"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:04.503307 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.501956 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95e2263e-c273-4505-baf4-65edea02ecd1-istiod-ca-cert" pod "95e2263e-c273-4505-baf4-65edea02ecd1" (UID: "95e2263e-c273-4505-baf4-65edea02ecd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:20:04.503307 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.502776 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-envoy" pod "4df160f4-4ae0-4d2f-8464-5a7e0e792434" (UID: "4df160f4-4ae0-4d2f-8464-5a7e0e792434"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:04.503307 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.503077 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-credential-socket" pod "95e2263e-c273-4505-baf4-65edea02ecd1" (UID: "95e2263e-c273-4505-baf4-65edea02ecd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:04.503658 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.503520 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e2263e-c273-4505-baf4-65edea02ecd1-kube-api-access-n5fcr" pod "95e2263e-c273-4505-baf4-65edea02ecd1" (UID: "95e2263e-c273-4505-baf4-65edea02ecd1"). InnerVolumeSpecName "kube-api-access-n5fcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:20:04.503658 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.503647 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-istio-envoy" pod "95e2263e-c273-4505-baf4-65edea02ecd1" (UID: "95e2263e-c273-4505-baf4-65edea02ecd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:04.503763 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.503650 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e2263e-c273-4505-baf4-65edea02ecd1-istio-token" pod "95e2263e-c273-4505-baf4-65edea02ecd1" (UID: "95e2263e-c273-4505-baf4-65edea02ecd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:20:04.503850 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.503819 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95e2263e-c273-4505-baf4-65edea02ecd1-istio-podinfo" pod "95e2263e-c273-4505-baf4-65edea02ecd1" (UID: "95e2263e-c273-4505-baf4-65edea02ecd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:20:04.504436 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.504412 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-token" pod "4df160f4-4ae0-4d2f-8464-5a7e0e792434" (UID: "4df160f4-4ae0-4d2f-8464-5a7e0e792434"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:20:04.504879 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.504858 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df160f4-4ae0-4d2f-8464-5a7e0e792434-kube-api-access-fmjz6" pod "4df160f4-4ae0-4d2f-8464-5a7e0e792434" (UID: "4df160f4-4ae0-4d2f-8464-5a7e0e792434"). InnerVolumeSpecName "kube-api-access-fmjz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:20:04.505106 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.505082 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-podinfo" pod "4df160f4-4ae0-4d2f-8464-5a7e0e792434" (UID: "4df160f4-4ae0-4d2f-8464-5a7e0e792434"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:20:04.602825 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.602790 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95e2263e-c273-4505-baf4-65edea02ecd1-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:04.602825 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.602824 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:04.602825 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.602834 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:04.603143 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.602843 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5fcr\" (UniqueName: \"kubernetes.io/projected/95e2263e-c273-4505-baf4-65edea02ecd1-kube-api-access-n5fcr\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:04.603143 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.602852 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:04.603143 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.602860 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:04.603143 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.602868 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:04.603143 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.602877 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:04.603143 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.602891 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e2263e-c273-4505-baf4-65edea02ecd1-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:04.603143 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.602904 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95e2263e-c273-4505-baf4-65edea02ecd1-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:04.603143 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.602912 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmjz6\" (UniqueName: \"kubernetes.io/projected/4df160f4-4ae0-4d2f-8464-5a7e0e792434-kube-api-access-fmjz6\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:04.603143 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.602920 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:04.603143 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.602928 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95e2263e-c273-4505-baf4-65edea02ecd1-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:04.603143 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.602936 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:04.603143 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:04.602944 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df160f4-4ae0-4d2f-8464-5a7e0e792434-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:05.413445 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:05.413348 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" Nov 02 23:20:05.413445 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:05.413357 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4" event={"ID":"95e2263e-c273-4505-baf4-65edea02ecd1","Type":"ContainerDied","Data":"cc29a41285525f4add8cff4a33cddc293ba478a8b64f83eb6dd6c7dbdc2fd629"} Nov 02 23:20:05.414037 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:05.413358 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d" Nov 02 23:20:05.414037 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:05.413459 5114 scope.go:122] "RemoveContainer" containerID="3cfd00ed5b081a10f19b01b68046d16ccc9d45085cdcca6f78ff81bea5f09684" Nov 02 23:20:05.693198 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:05.693106 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4"] Nov 02 23:20:05.742816 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:05.742781 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-bt7v4"] Nov 02 23:20:05.806555 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:05.806519 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d"] Nov 02 23:20:05.810145 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:05.810111 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-lj67d"] Nov 02 23:20:06.794028 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:06.793992 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4df160f4-4ae0-4d2f-8464-5a7e0e792434" path="/var/lib/kubelet/pods/4df160f4-4ae0-4d2f-8464-5a7e0e792434/volumes" Nov 02 23:20:06.794561 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:06.794545 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95e2263e-c273-4505-baf4-65edea02ecd1" path="/var/lib/kubelet/pods/95e2263e-c273-4505-baf4-65edea02ecd1/volumes" Nov 02 23:20:07.584532 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:20:07.584476 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:20:08.304157 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:08.304122 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds"] Nov 02 23:20:08.304648 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:08.304363 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" podUID="0879a805-4cb8-42a1-99f4-0e18d318335f" containerName="istio-proxy" containerID="cri-o://af3b1c9186e95266421c013f47c408d17399e582e267950b726a1bf86d89a0b6" gracePeriod=30 Nov 02 23:20:10.046160 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:10.046123 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dea36f3-f4r92"] Nov 02 23:20:10.095386 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:10.095352 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dea36f3-f4r92"] Nov 02 23:20:10.794203 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:10.794164 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="033ac52a-46b6-4b42-b9c0-b5f31a6b1675" path="/var/lib/kubelet/pods/033ac52a-46b6-4b42-b9c0-b5f31a6b1675/volumes" Nov 02 23:20:13.460186 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.460154 5114 generic.go:356] "Generic (PLEG): container finished" podID="0879a805-4cb8-42a1-99f4-0e18d318335f" containerID="af3b1c9186e95266421c013f47c408d17399e582e267950b726a1bf86d89a0b6" exitCode=0 Nov 02 23:20:13.460644 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.460328 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" event={"ID":"0879a805-4cb8-42a1-99f4-0e18d318335f","Type":"ContainerDied","Data":"af3b1c9186e95266421c013f47c408d17399e582e267950b726a1bf86d89a0b6"} Nov 02 23:20:13.580335 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.580252 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:20:13.634011 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.633979 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-workload-socket\") pod \"0879a805-4cb8-42a1-99f4-0e18d318335f\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " Nov 02 23:20:13.634200 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.634043 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0879a805-4cb8-42a1-99f4-0e18d318335f-kube-api-access-9thqj\" (UniqueName: \"kubernetes.io/projected/0879a805-4cb8-42a1-99f4-0e18d318335f-kube-api-access-9thqj\") pod \"0879a805-4cb8-42a1-99f4-0e18d318335f\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " Nov 02 23:20:13.634200 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.634066 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-workload-certs\") pod \"0879a805-4cb8-42a1-99f4-0e18d318335f\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " Nov 02 23:20:13.634200 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.634104 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-podinfo\") pod \"0879a805-4cb8-42a1-99f4-0e18d318335f\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " Nov 02 23:20:13.634200 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.634142 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-envoy\") pod \"0879a805-4cb8-42a1-99f4-0e18d318335f\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " Nov 02 23:20:13.634200 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.634186 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-credential-socket\") pod \"0879a805-4cb8-42a1-99f4-0e18d318335f\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " Nov 02 23:20:13.634471 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.634230 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-token\" (UniqueName: \"kubernetes.io/projected/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-token\") pod \"0879a805-4cb8-42a1-99f4-0e18d318335f\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " Nov 02 23:20:13.634471 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.634263 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-data\") pod \"0879a805-4cb8-42a1-99f4-0e18d318335f\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " Nov 02 23:20:13.634471 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.634306 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0879a805-4cb8-42a1-99f4-0e18d318335f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0879a805-4cb8-42a1-99f4-0e18d318335f-istiod-ca-cert\") pod \"0879a805-4cb8-42a1-99f4-0e18d318335f\" (UID: \"0879a805-4cb8-42a1-99f4-0e18d318335f\") " Nov 02 23:20:13.634471 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.634315 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-workload-socket" pod "0879a805-4cb8-42a1-99f4-0e18d318335f" (UID: "0879a805-4cb8-42a1-99f4-0e18d318335f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:13.634700 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.634685 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:13.635008 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.634984 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-workload-certs" pod "0879a805-4cb8-42a1-99f4-0e18d318335f" (UID: "0879a805-4cb8-42a1-99f4-0e18d318335f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:13.635203 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.635176 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0879a805-4cb8-42a1-99f4-0e18d318335f-istiod-ca-cert" pod "0879a805-4cb8-42a1-99f4-0e18d318335f" (UID: "0879a805-4cb8-42a1-99f4-0e18d318335f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:20:13.635496 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.635361 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-credential-socket" pod "0879a805-4cb8-42a1-99f4-0e18d318335f" (UID: "0879a805-4cb8-42a1-99f4-0e18d318335f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:13.637013 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.636966 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-envoy" pod "0879a805-4cb8-42a1-99f4-0e18d318335f" (UID: "0879a805-4cb8-42a1-99f4-0e18d318335f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:13.637013 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.636989 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-data" pod "0879a805-4cb8-42a1-99f4-0e18d318335f" (UID: "0879a805-4cb8-42a1-99f4-0e18d318335f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:13.637171 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.637141 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0879a805-4cb8-42a1-99f4-0e18d318335f-kube-api-access-9thqj" pod "0879a805-4cb8-42a1-99f4-0e18d318335f" (UID: "0879a805-4cb8-42a1-99f4-0e18d318335f"). InnerVolumeSpecName "kube-api-access-9thqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:20:13.637374 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.637346 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-token" pod "0879a805-4cb8-42a1-99f4-0e18d318335f" (UID: "0879a805-4cb8-42a1-99f4-0e18d318335f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:20:13.638224 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.638188 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-podinfo" pod "0879a805-4cb8-42a1-99f4-0e18d318335f" (UID: "0879a805-4cb8-42a1-99f4-0e18d318335f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:20:13.736295 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.736256 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0879a805-4cb8-42a1-99f4-0e18d318335f-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:13.736295 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.736293 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9thqj\" (UniqueName: \"kubernetes.io/projected/0879a805-4cb8-42a1-99f4-0e18d318335f-kube-api-access-9thqj\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:13.736295 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.736304 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:13.736606 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.736315 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:13.736606 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.736324 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:13.736606 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.736333 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:13.736606 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.736341 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:13.736606 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:13.736349 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0879a805-4cb8-42a1-99f4-0e18d318335f-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:14.467698 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:14.467657 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" event={"ID":"0879a805-4cb8-42a1-99f4-0e18d318335f","Type":"ContainerDied","Data":"f28d8db9028d6fed2f67d62bc6614adc11231013e6ede1aeb78b4f4e01c6513c"} Nov 02 23:20:14.467698 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:14.467679 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds" Nov 02 23:20:14.468222 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:14.467732 5114 scope.go:122] "RemoveContainer" containerID="af3b1c9186e95266421c013f47c408d17399e582e267950b726a1bf86d89a0b6" Nov 02 23:20:14.683606 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:14.683569 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds"] Nov 02 23:20:14.687348 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:14.687316 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-4xrds"] Nov 02 23:20:14.793778 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:14.793742 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0879a805-4cb8-42a1-99f4-0e18d318335f" path="/var/lib/kubelet/pods/0879a805-4cb8-42a1-99f4-0e18d318335f/volumes" Nov 02 23:20:37.584403 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:20:37.584319 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:20:45.122026 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:20:45.121986 5114 scope.go:122] "RemoveContainer" containerID="9d418505c24e6abab1e4b7a6c5ff84eb0374134b67844c3b8fc60b8833f96465" Nov 02 23:21:07.584851 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:21:07.584811 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:21:37.586207 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:21:37.586168 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:21:40.416659 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:40.416620 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh"] Nov 02 23:21:40.417179 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:40.416885 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" podUID="13beeff1-dd93-473c-b8c8-55ff28a8d746" containerName="istio-proxy" containerID="cri-o://1fd295d2b5085b486f32f44dc83899206c1a24d768d84a6013d0374793e112be" gracePeriod=30 Nov 02 23:21:41.124974 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:41.124937 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:21:41.125166 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:41.125008 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" podUID="13beeff1-dd93-473c-b8c8-55ff28a8d746" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:21:45.153745 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.153710 5114 scope.go:122] "RemoveContainer" containerID="6c2a94a88907463d9ba019cd5e1acd3a783faa6322ac95ffc29a40eca585ed3b" Nov 02 23:21:45.677947 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.677918 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:21:45.787901 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.787850 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-workload-socket\") pod \"13beeff1-dd93-473c-b8c8-55ff28a8d746\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " Nov 02 23:21:45.788129 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.787925 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-workload-certs\") pod \"13beeff1-dd93-473c-b8c8-55ff28a8d746\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " Nov 02 23:21:45.788129 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.787977 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-podinfo\") pod \"13beeff1-dd93-473c-b8c8-55ff28a8d746\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " Nov 02 23:21:45.788129 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.788027 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-data\") pod \"13beeff1-dd93-473c-b8c8-55ff28a8d746\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " Nov 02 23:21:45.788129 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.788072 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-envoy\") pod \"13beeff1-dd93-473c-b8c8-55ff28a8d746\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " Nov 02 23:21:45.788357 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.788142 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13beeff1-dd93-473c-b8c8-55ff28a8d746-kube-api-access-qsttz\" (UniqueName: \"kubernetes.io/projected/13beeff1-dd93-473c-b8c8-55ff28a8d746-kube-api-access-qsttz\") pod \"13beeff1-dd93-473c-b8c8-55ff28a8d746\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " Nov 02 23:21:45.788357 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.788202 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-token\" (UniqueName: \"kubernetes.io/projected/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-token\") pod \"13beeff1-dd93-473c-b8c8-55ff28a8d746\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " Nov 02 23:21:45.788357 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.788248 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-credential-socket\") pod \"13beeff1-dd93-473c-b8c8-55ff28a8d746\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " Nov 02 23:21:45.788357 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.788263 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-workload-socket" pod "13beeff1-dd93-473c-b8c8-55ff28a8d746" (UID: "13beeff1-dd93-473c-b8c8-55ff28a8d746"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:21:45.788357 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.788284 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13beeff1-dd93-473c-b8c8-55ff28a8d746-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13beeff1-dd93-473c-b8c8-55ff28a8d746-istiod-ca-cert\") pod \"13beeff1-dd93-473c-b8c8-55ff28a8d746\" (UID: \"13beeff1-dd93-473c-b8c8-55ff28a8d746\") " Nov 02 23:21:45.788357 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.788336 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-workload-certs" pod "13beeff1-dd93-473c-b8c8-55ff28a8d746" (UID: "13beeff1-dd93-473c-b8c8-55ff28a8d746"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:21:45.788690 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.788601 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13beeff1-dd93-473c-b8c8-55ff28a8d746-istiod-ca-cert" pod "13beeff1-dd93-473c-b8c8-55ff28a8d746" (UID: "13beeff1-dd93-473c-b8c8-55ff28a8d746"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:21:45.788966 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.788937 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13beeff1-dd93-473c-b8c8-55ff28a8d746-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:45.789132 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.788973 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:45.789132 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.788992 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:45.789671 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.789588 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-credential-socket" pod "13beeff1-dd93-473c-b8c8-55ff28a8d746" (UID: "13beeff1-dd93-473c-b8c8-55ff28a8d746"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:21:45.789671 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.789630 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-data" pod "13beeff1-dd93-473c-b8c8-55ff28a8d746" (UID: "13beeff1-dd93-473c-b8c8-55ff28a8d746"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:21:45.791411 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.791363 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-envoy" pod "13beeff1-dd93-473c-b8c8-55ff28a8d746" (UID: "13beeff1-dd93-473c-b8c8-55ff28a8d746"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:21:45.791873 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.791840 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13beeff1-dd93-473c-b8c8-55ff28a8d746-kube-api-access-qsttz" pod "13beeff1-dd93-473c-b8c8-55ff28a8d746" (UID: "13beeff1-dd93-473c-b8c8-55ff28a8d746"). InnerVolumeSpecName "kube-api-access-qsttz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:21:45.792019 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.791988 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-token" pod "13beeff1-dd93-473c-b8c8-55ff28a8d746" (UID: "13beeff1-dd93-473c-b8c8-55ff28a8d746"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:21:45.792196 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.792169 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-podinfo" pod "13beeff1-dd93-473c-b8c8-55ff28a8d746" (UID: "13beeff1-dd93-473c-b8c8-55ff28a8d746"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:21:45.890283 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.890227 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:45.890283 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.890276 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:45.890283 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.890291 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:45.890652 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.890305 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:45.890652 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.890319 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13beeff1-dd93-473c-b8c8-55ff28a8d746-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:45.890652 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:45.890333 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsttz\" (UniqueName: \"kubernetes.io/projected/13beeff1-dd93-473c-b8c8-55ff28a8d746-kube-api-access-qsttz\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:46.005820 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:46.005788 5114 generic.go:356] "Generic (PLEG): container finished" podID="13beeff1-dd93-473c-b8c8-55ff28a8d746" containerID="1fd295d2b5085b486f32f44dc83899206c1a24d768d84a6013d0374793e112be" exitCode=0 Nov 02 23:21:46.006000 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:46.005836 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" Nov 02 23:21:46.006000 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:46.005865 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" event={"ID":"13beeff1-dd93-473c-b8c8-55ff28a8d746","Type":"ContainerDied","Data":"1fd295d2b5085b486f32f44dc83899206c1a24d768d84a6013d0374793e112be"} Nov 02 23:21:46.006000 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:46.005922 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh" event={"ID":"13beeff1-dd93-473c-b8c8-55ff28a8d746","Type":"ContainerDied","Data":"c736aa104ff741d1ebbce7b61f6adbdae6755da938df51e620022c5fd46dc097"} Nov 02 23:21:46.006000 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:46.005945 5114 scope.go:122] "RemoveContainer" containerID="1fd295d2b5085b486f32f44dc83899206c1a24d768d84a6013d0374793e112be" Nov 02 23:21:46.016708 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:46.016688 5114 scope.go:122] "RemoveContainer" containerID="1fd295d2b5085b486f32f44dc83899206c1a24d768d84a6013d0374793e112be" Nov 02 23:21:46.017074 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:21:46.017047 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fd295d2b5085b486f32f44dc83899206c1a24d768d84a6013d0374793e112be\": container with ID starting with 1fd295d2b5085b486f32f44dc83899206c1a24d768d84a6013d0374793e112be not found: ID does not exist" containerID="1fd295d2b5085b486f32f44dc83899206c1a24d768d84a6013d0374793e112be" Nov 02 23:21:46.017162 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:46.017084 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd295d2b5085b486f32f44dc83899206c1a24d768d84a6013d0374793e112be"} err="failed to get container status \"1fd295d2b5085b486f32f44dc83899206c1a24d768d84a6013d0374793e112be\": rpc error: code = NotFound desc = could not find container \"1fd295d2b5085b486f32f44dc83899206c1a24d768d84a6013d0374793e112be\": container with ID starting with 1fd295d2b5085b486f32f44dc83899206c1a24d768d84a6013d0374793e112be not found: ID does not exist" Nov 02 23:21:46.169966 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:46.169878 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh"] Nov 02 23:21:46.174806 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:46.174769 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-8s2wh"] Nov 02 23:21:46.794364 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:21:46.794316 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13beeff1-dd93-473c-b8c8-55ff28a8d746" path="/var/lib/kubelet/pods/13beeff1-dd93-473c-b8c8-55ff28a8d746/volumes" Nov 02 23:22:02.319965 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:02.319920 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv"] Nov 02 23:22:02.320443 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:02.320136 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" podUID="151a9d61-9d15-403d-955a-c398c7bec160" containerName="istio-proxy" containerID="cri-o://0a11f6b86f5dcf550af8206d7741a883839e4c212d8a52319aeac931d22f70a1" gracePeriod=30 Nov 02 23:22:06.170409 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:06.170366 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-hlxtv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:22:06.170856 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:06.170438 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" podUID="151a9d61-9d15-403d-955a-c398c7bec160" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:22:07.584822 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.584792 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:22:07.590674 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:22:07.590571 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:22:07.748036 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.748001 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151a9d61-9d15-403d-955a-c398c7bec160-istio-token\" (UniqueName: \"kubernetes.io/projected/151a9d61-9d15-403d-955a-c398c7bec160-istio-token\") pod \"151a9d61-9d15-403d-955a-c398c7bec160\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " Nov 02 23:22:07.748234 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.748094 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/151a9d61-9d15-403d-955a-c398c7bec160-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151a9d61-9d15-403d-955a-c398c7bec160-istio-podinfo\") pod \"151a9d61-9d15-403d-955a-c398c7bec160\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " Nov 02 23:22:07.748234 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.748117 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-workload-certs\") pod \"151a9d61-9d15-403d-955a-c398c7bec160\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " Nov 02 23:22:07.748234 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.748140 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-istio-envoy\") pod \"151a9d61-9d15-403d-955a-c398c7bec160\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " Nov 02 23:22:07.748234 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.748172 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-workload-socket\") pod \"151a9d61-9d15-403d-955a-c398c7bec160\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " Nov 02 23:22:07.748234 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.748210 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/151a9d61-9d15-403d-955a-c398c7bec160-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151a9d61-9d15-403d-955a-c398c7bec160-istiod-ca-cert\") pod \"151a9d61-9d15-403d-955a-c398c7bec160\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " Nov 02 23:22:07.748539 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.748265 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151a9d61-9d15-403d-955a-c398c7bec160-kube-api-access-887kk\" (UniqueName: \"kubernetes.io/projected/151a9d61-9d15-403d-955a-c398c7bec160-kube-api-access-887kk\") pod \"151a9d61-9d15-403d-955a-c398c7bec160\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " Nov 02 23:22:07.748539 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.748399 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-credential-socket\") pod \"151a9d61-9d15-403d-955a-c398c7bec160\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " Nov 02 23:22:07.748539 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.748451 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-istio-data\") pod \"151a9d61-9d15-403d-955a-c398c7bec160\" (UID: \"151a9d61-9d15-403d-955a-c398c7bec160\") " Nov 02 23:22:07.748539 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.748455 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-workload-certs" pod "151a9d61-9d15-403d-955a-c398c7bec160" (UID: "151a9d61-9d15-403d-955a-c398c7bec160"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:07.748762 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.748514 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-workload-socket" pod "151a9d61-9d15-403d-955a-c398c7bec160" (UID: "151a9d61-9d15-403d-955a-c398c7bec160"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:07.748762 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.748730 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-credential-socket" pod "151a9d61-9d15-403d-955a-c398c7bec160" (UID: "151a9d61-9d15-403d-955a-c398c7bec160"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:07.749040 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.748844 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-istio-data" pod "151a9d61-9d15-403d-955a-c398c7bec160" (UID: "151a9d61-9d15-403d-955a-c398c7bec160"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:07.749179 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.749122 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/151a9d61-9d15-403d-955a-c398c7bec160-istiod-ca-cert" pod "151a9d61-9d15-403d-955a-c398c7bec160" (UID: "151a9d61-9d15-403d-955a-c398c7bec160"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:22:07.749514 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.749472 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:07.749625 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.749521 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:07.749625 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.749537 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151a9d61-9d15-403d-955a-c398c7bec160-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:07.749625 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.749551 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:07.749625 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.749567 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:07.750645 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.750612 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/151a9d61-9d15-403d-955a-c398c7bec160-istio-podinfo" pod "151a9d61-9d15-403d-955a-c398c7bec160" (UID: "151a9d61-9d15-403d-955a-c398c7bec160"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:22:07.750772 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.750655 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-istio-envoy" pod "151a9d61-9d15-403d-955a-c398c7bec160" (UID: "151a9d61-9d15-403d-955a-c398c7bec160"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:07.750772 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.750704 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151a9d61-9d15-403d-955a-c398c7bec160-istio-token" pod "151a9d61-9d15-403d-955a-c398c7bec160" (UID: "151a9d61-9d15-403d-955a-c398c7bec160"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:07.750900 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.750876 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151a9d61-9d15-403d-955a-c398c7bec160-kube-api-access-887kk" pod "151a9d61-9d15-403d-955a-c398c7bec160" (UID: "151a9d61-9d15-403d-955a-c398c7bec160"). InnerVolumeSpecName "kube-api-access-887kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:07.850838 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.850729 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/151a9d61-9d15-403d-955a-c398c7bec160-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:07.850838 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.850780 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151a9d61-9d15-403d-955a-c398c7bec160-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:07.850838 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.850790 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151a9d61-9d15-403d-955a-c398c7bec160-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:07.850838 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:07.850800 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-887kk\" (UniqueName: \"kubernetes.io/projected/151a9d61-9d15-403d-955a-c398c7bec160-kube-api-access-887kk\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:08.129932 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:08.129839 5114 generic.go:356] "Generic (PLEG): container finished" podID="151a9d61-9d15-403d-955a-c398c7bec160" containerID="0a11f6b86f5dcf550af8206d7741a883839e4c212d8a52319aeac931d22f70a1" exitCode=0 Nov 02 23:22:08.129932 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:08.129890 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" Nov 02 23:22:08.130157 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:08.129933 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" event={"ID":"151a9d61-9d15-403d-955a-c398c7bec160","Type":"ContainerDied","Data":"0a11f6b86f5dcf550af8206d7741a883839e4c212d8a52319aeac931d22f70a1"} Nov 02 23:22:08.130157 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:08.129993 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv" event={"ID":"151a9d61-9d15-403d-955a-c398c7bec160","Type":"ContainerDied","Data":"58f7457ec21b550be03c6f600c1f9650cbafbaa5f6f12eac4a699497e680194b"} Nov 02 23:22:08.130157 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:08.130016 5114 scope.go:122] "RemoveContainer" containerID="0a11f6b86f5dcf550af8206d7741a883839e4c212d8a52319aeac931d22f70a1" Nov 02 23:22:08.143186 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:08.143159 5114 scope.go:122] "RemoveContainer" containerID="0a11f6b86f5dcf550af8206d7741a883839e4c212d8a52319aeac931d22f70a1" Nov 02 23:22:08.143565 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:22:08.143545 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a11f6b86f5dcf550af8206d7741a883839e4c212d8a52319aeac931d22f70a1\": container with ID starting with 0a11f6b86f5dcf550af8206d7741a883839e4c212d8a52319aeac931d22f70a1 not found: ID does not exist" containerID="0a11f6b86f5dcf550af8206d7741a883839e4c212d8a52319aeac931d22f70a1" Nov 02 23:22:08.143622 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:08.143576 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a11f6b86f5dcf550af8206d7741a883839e4c212d8a52319aeac931d22f70a1"} err="failed to get container status \"0a11f6b86f5dcf550af8206d7741a883839e4c212d8a52319aeac931d22f70a1\": rpc error: code = NotFound desc = could not find container \"0a11f6b86f5dcf550af8206d7741a883839e4c212d8a52319aeac931d22f70a1\": container with ID starting with 0a11f6b86f5dcf550af8206d7741a883839e4c212d8a52319aeac931d22f70a1 not found: ID does not exist" Nov 02 23:22:08.294742 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:08.294706 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv"] Nov 02 23:22:08.344225 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:08.344185 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hlxtv"] Nov 02 23:22:08.794290 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:22:08.794246 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="151a9d61-9d15-403d-955a-c398c7bec160" path="/var/lib/kubelet/pods/151a9d61-9d15-403d-955a-c398c7bec160/volumes" Nov 02 23:22:37.585209 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:22:37.585163 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:23:07.584743 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:23:07.584696 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:23:37.584349 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:23:37.584252 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:24:07.584372 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:24:07.584329 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:24:37.585025 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:24:37.584976 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:25:07.584397 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:25:07.584356 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:25:26.250999 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.250959 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl"] Nov 02 23:25:26.251589 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.251238 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" podUID="a4da1367-063f-4955-9313-26c9e7b97729" containerName="discovery" containerID="cri-o://25de7f71386fefb8b10519afcf85d920550909664a601e38d01978930a28511e" gracePeriod=30 Nov 02 23:25:26.419944 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.419905 5114 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-zr4hl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:8080/ready\": dial tcp 10.134.0.61:8080: connect: connection refused" start-of-body= Nov 02 23:25:26.420146 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.419965 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" podUID="a4da1367-063f-4955-9313-26c9e7b97729" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.61:8080/ready\": dial tcp 10.134.0.61:8080: connect: connection refused" Nov 02 23:25:26.525439 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.525410 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:25:26.568590 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.568552 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4da1367-063f-4955-9313-26c9e7b97729-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a4da1367-063f-4955-9313-26c9e7b97729-istio-csr-ca-configmap\") pod \"a4da1367-063f-4955-9313-26c9e7b97729\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " Nov 02 23:25:26.568800 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.568600 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4da1367-063f-4955-9313-26c9e7b97729-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a4da1367-063f-4955-9313-26c9e7b97729-istio-kubeconfig\") pod \"a4da1367-063f-4955-9313-26c9e7b97729\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " Nov 02 23:25:26.568800 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.568673 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4da1367-063f-4955-9313-26c9e7b97729-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4da1367-063f-4955-9313-26c9e7b97729-local-certs\") pod \"a4da1367-063f-4955-9313-26c9e7b97729\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " Nov 02 23:25:26.568800 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.568713 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4da1367-063f-4955-9313-26c9e7b97729-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a4da1367-063f-4955-9313-26c9e7b97729-istio-csr-dns-cert\") pod \"a4da1367-063f-4955-9313-26c9e7b97729\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " Nov 02 23:25:26.568800 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.568769 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4da1367-063f-4955-9313-26c9e7b97729-istio-token\" (UniqueName: \"kubernetes.io/projected/a4da1367-063f-4955-9313-26c9e7b97729-istio-token\") pod \"a4da1367-063f-4955-9313-26c9e7b97729\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " Nov 02 23:25:26.568800 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.568792 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4da1367-063f-4955-9313-26c9e7b97729-cacerts\" (UniqueName: \"kubernetes.io/secret/a4da1367-063f-4955-9313-26c9e7b97729-cacerts\") pod \"a4da1367-063f-4955-9313-26c9e7b97729\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " Nov 02 23:25:26.569083 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.568818 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4da1367-063f-4955-9313-26c9e7b97729-kube-api-access-5d4vm\" (UniqueName: \"kubernetes.io/projected/a4da1367-063f-4955-9313-26c9e7b97729-kube-api-access-5d4vm\") pod \"a4da1367-063f-4955-9313-26c9e7b97729\" (UID: \"a4da1367-063f-4955-9313-26c9e7b97729\") " Nov 02 23:25:26.569083 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.569025 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4da1367-063f-4955-9313-26c9e7b97729-istio-csr-ca-configmap" pod "a4da1367-063f-4955-9313-26c9e7b97729" (UID: "a4da1367-063f-4955-9313-26c9e7b97729"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:25:26.569375 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.569350 5114 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a4da1367-063f-4955-9313-26c9e7b97729-istio-csr-ca-configmap\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:26.571592 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.571559 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4da1367-063f-4955-9313-26c9e7b97729-kube-api-access-5d4vm" pod "a4da1367-063f-4955-9313-26c9e7b97729" (UID: "a4da1367-063f-4955-9313-26c9e7b97729"). InnerVolumeSpecName "kube-api-access-5d4vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:26.571701 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.571643 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4da1367-063f-4955-9313-26c9e7b97729-istio-csr-dns-cert" pod "a4da1367-063f-4955-9313-26c9e7b97729" (UID: "a4da1367-063f-4955-9313-26c9e7b97729"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:25:26.572067 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.572039 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4da1367-063f-4955-9313-26c9e7b97729-istio-kubeconfig" pod "a4da1367-063f-4955-9313-26c9e7b97729" (UID: "a4da1367-063f-4955-9313-26c9e7b97729"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:25:26.572067 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.572048 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4da1367-063f-4955-9313-26c9e7b97729-istio-token" pod "a4da1367-063f-4955-9313-26c9e7b97729" (UID: "a4da1367-063f-4955-9313-26c9e7b97729"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:26.572215 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.572098 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4da1367-063f-4955-9313-26c9e7b97729-cacerts" pod "a4da1367-063f-4955-9313-26c9e7b97729" (UID: "a4da1367-063f-4955-9313-26c9e7b97729"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:25:26.572215 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.572169 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4da1367-063f-4955-9313-26c9e7b97729-local-certs" pod "a4da1367-063f-4955-9313-26c9e7b97729" (UID: "a4da1367-063f-4955-9313-26c9e7b97729"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:26.670400 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.670350 5114 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a4da1367-063f-4955-9313-26c9e7b97729-istio-kubeconfig\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:26.670400 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.670390 5114 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4da1367-063f-4955-9313-26c9e7b97729-local-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:26.670400 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.670405 5114 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a4da1367-063f-4955-9313-26c9e7b97729-istio-csr-dns-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:26.670668 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.670417 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4da1367-063f-4955-9313-26c9e7b97729-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:26.670668 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.670429 5114 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a4da1367-063f-4955-9313-26c9e7b97729-cacerts\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:26.670668 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:26.670442 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d4vm\" (UniqueName: \"kubernetes.io/projected/a4da1367-063f-4955-9313-26c9e7b97729-kube-api-access-5d4vm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:27.305337 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:27.305300 5114 generic.go:356] "Generic (PLEG): container finished" podID="a4da1367-063f-4955-9313-26c9e7b97729" containerID="25de7f71386fefb8b10519afcf85d920550909664a601e38d01978930a28511e" exitCode=0 Nov 02 23:25:27.305874 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:27.305411 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" event={"ID":"a4da1367-063f-4955-9313-26c9e7b97729","Type":"ContainerDied","Data":"25de7f71386fefb8b10519afcf85d920550909664a601e38d01978930a28511e"} Nov 02 23:25:27.305874 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:27.305519 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" event={"ID":"a4da1367-063f-4955-9313-26c9e7b97729","Type":"ContainerDied","Data":"41b7a4de3e1554dcbf3f737ac6a27223be383d86091ac421e08d194d1b061f80"} Nov 02 23:25:27.305874 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:27.305541 5114 scope.go:122] "RemoveContainer" containerID="25de7f71386fefb8b10519afcf85d920550909664a601e38d01978930a28511e" Nov 02 23:25:27.305874 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:27.305621 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl" Nov 02 23:25:27.317383 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:27.317353 5114 scope.go:122] "RemoveContainer" containerID="25de7f71386fefb8b10519afcf85d920550909664a601e38d01978930a28511e" Nov 02 23:25:27.317775 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:25:27.317745 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25de7f71386fefb8b10519afcf85d920550909664a601e38d01978930a28511e\": container with ID starting with 25de7f71386fefb8b10519afcf85d920550909664a601e38d01978930a28511e not found: ID does not exist" containerID="25de7f71386fefb8b10519afcf85d920550909664a601e38d01978930a28511e" Nov 02 23:25:27.317873 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:27.317786 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25de7f71386fefb8b10519afcf85d920550909664a601e38d01978930a28511e"} err="failed to get container status \"25de7f71386fefb8b10519afcf85d920550909664a601e38d01978930a28511e\": rpc error: code = NotFound desc = could not find container \"25de7f71386fefb8b10519afcf85d920550909664a601e38d01978930a28511e\": container with ID starting with 25de7f71386fefb8b10519afcf85d920550909664a601e38d01978930a28511e not found: ID does not exist" Nov 02 23:25:27.469758 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:27.469721 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl"] Nov 02 23:25:27.518812 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:27.518775 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zr4hl"] Nov 02 23:25:28.792792 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:28.792753 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4da1367-063f-4955-9313-26c9e7b97729" path="/var/lib/kubelet/pods/a4da1367-063f-4955-9313-26c9e7b97729/volumes" Nov 02 23:25:35.698826 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.698778 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h"] Nov 02 23:25:35.699463 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699438 5114 cpu_manager.go:414] "containerMap: removing container" podUID="fa02eae2-80f1-4a52-a122-1f2451ea4ebd" containerName="revoke-keys" Nov 02 23:25:35.699575 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699464 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa02eae2-80f1-4a52-a122-1f2451ea4ebd" containerName="revoke-keys" Nov 02 23:25:35.699575 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699499 5114 cpu_manager.go:414] "containerMap: removing container" podUID="a4da1367-063f-4955-9313-26c9e7b97729" containerName="discovery" Nov 02 23:25:35.699575 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699508 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4da1367-063f-4955-9313-26c9e7b97729" containerName="discovery" Nov 02 23:25:35.699575 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699527 5114 cpu_manager.go:414] "containerMap: removing container" podUID="0879a805-4cb8-42a1-99f4-0e18d318335f" containerName="istio-proxy" Nov 02 23:25:35.699575 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699534 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0879a805-4cb8-42a1-99f4-0e18d318335f" containerName="istio-proxy" Nov 02 23:25:35.699575 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699552 5114 cpu_manager.go:414] "containerMap: removing container" podUID="13beeff1-dd93-473c-b8c8-55ff28a8d746" containerName="istio-proxy" Nov 02 23:25:35.699575 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699559 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13beeff1-dd93-473c-b8c8-55ff28a8d746" containerName="istio-proxy" Nov 02 23:25:35.699920 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699588 5114 cpu_manager.go:414] "containerMap: removing container" podUID="4df160f4-4ae0-4d2f-8464-5a7e0e792434" containerName="istio-proxy" Nov 02 23:25:35.699920 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699595 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4df160f4-4ae0-4d2f-8464-5a7e0e792434" containerName="istio-proxy" Nov 02 23:25:35.699920 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699609 5114 cpu_manager.go:414] "containerMap: removing container" podUID="3a1bf336-e967-4582-83b8-12584603fd63" containerName="cleanup" Nov 02 23:25:35.699920 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699616 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1bf336-e967-4582-83b8-12584603fd63" containerName="cleanup" Nov 02 23:25:35.699920 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699627 5114 cpu_manager.go:414] "containerMap: removing container" podUID="95e2263e-c273-4505-baf4-65edea02ecd1" containerName="istio-proxy" Nov 02 23:25:35.699920 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699634 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95e2263e-c273-4505-baf4-65edea02ecd1" containerName="istio-proxy" Nov 02 23:25:35.699920 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699646 5114 cpu_manager.go:414] "containerMap: removing container" podUID="6b3a4d68-657c-4da3-8fbf-891f0926b134" containerName="cleanup" Nov 02 23:25:35.699920 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699654 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3a4d68-657c-4da3-8fbf-891f0926b134" containerName="cleanup" Nov 02 23:25:35.699920 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699664 5114 cpu_manager.go:414] "containerMap: removing container" podUID="151a9d61-9d15-403d-955a-c398c7bec160" containerName="istio-proxy" Nov 02 23:25:35.699920 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699671 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151a9d61-9d15-403d-955a-c398c7bec160" containerName="istio-proxy" Nov 02 23:25:35.699920 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699786 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="4df160f4-4ae0-4d2f-8464-5a7e0e792434" containerName="istio-proxy" Nov 02 23:25:35.699920 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699802 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1bf336-e967-4582-83b8-12584603fd63" containerName="cleanup" Nov 02 23:25:35.699920 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699812 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="95e2263e-c273-4505-baf4-65edea02ecd1" containerName="istio-proxy" Nov 02 23:25:35.699920 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699821 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="151a9d61-9d15-403d-955a-c398c7bec160" containerName="istio-proxy" Nov 02 23:25:35.699920 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699834 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa02eae2-80f1-4a52-a122-1f2451ea4ebd" containerName="revoke-keys" Nov 02 23:25:35.699920 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699844 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4da1367-063f-4955-9313-26c9e7b97729" containerName="discovery" Nov 02 23:25:35.699920 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699854 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="0879a805-4cb8-42a1-99f4-0e18d318335f" containerName="istio-proxy" Nov 02 23:25:35.699920 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.699863 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="13beeff1-dd93-473c-b8c8-55ff28a8d746" containerName="istio-proxy" Nov 02 23:25:35.704474 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.704446 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.762707 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.762666 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-85efc6-openshift-default-dockercfg-ppc6k\"" Nov 02 23:25:35.776302 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.776255 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2ff8\" (UniqueName: \"kubernetes.io/projected/336646a8-fbe7-4780-85b8-116a012ea309-kube-api-access-v2ff8\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.776680 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.776653 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-workload-certs\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.776839 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.776823 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-istio-data\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.777048 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.777028 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-workload-socket\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.777222 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.777206 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/336646a8-fbe7-4780-85b8-116a012ea309-istio-token\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.777347 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.777332 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-istio-envoy\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.777460 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.777447 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/336646a8-fbe7-4780-85b8-116a012ea309-istiod-ca-cert\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.777606 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.777591 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-credential-socket\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.777730 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.777714 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/336646a8-fbe7-4780-85b8-116a012ea309-istio-podinfo\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.830515 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.823070 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h"] Nov 02 23:25:35.900124 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.900067 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/336646a8-fbe7-4780-85b8-116a012ea309-istio-token\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.900341 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.900168 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-istio-envoy\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.900720 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.900692 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/336646a8-fbe7-4780-85b8-116a012ea309-istiod-ca-cert\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.901004 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.900979 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-credential-socket\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.901512 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.901473 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/336646a8-fbe7-4780-85b8-116a012ea309-istio-podinfo\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.901733 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.901406 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-credential-socket\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.901804 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.901290 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/336646a8-fbe7-4780-85b8-116a012ea309-istiod-ca-cert\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.901912 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.901893 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2ff8\" (UniqueName: \"kubernetes.io/projected/336646a8-fbe7-4780-85b8-116a012ea309-kube-api-access-v2ff8\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.902028 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.902014 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-workload-certs\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.902157 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.902143 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-istio-data\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.902358 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.902341 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-workload-socket\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.902473 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.902447 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-workload-certs\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.902775 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.902750 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-istio-data\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.903075 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.903049 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-workload-socket\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.903401 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.903379 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-istio-envoy\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.905723 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.905701 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/336646a8-fbe7-4780-85b8-116a012ea309-istio-podinfo\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.960842 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.960764 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/336646a8-fbe7-4780-85b8-116a012ea309-istio-token\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:35.964984 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:35.964953 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2ff8\" (UniqueName: \"kubernetes.io/projected/336646a8-fbe7-4780-85b8-116a012ea309-kube-api-access-v2ff8\") pod \"e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:36.018418 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:36.018383 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:36.465737 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:36.465705 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h"] Nov 02 23:25:36.468287 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:25:36.468250 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336646a8_fbe7_4780_85b8_116a012ea309.slice/crio-b6f19d0b627f95296c6bff087b2537c52c45226e040331d8ae29055ac62b1952 WatchSource:0}: Error finding container b6f19d0b627f95296c6bff087b2537c52c45226e040331d8ae29055ac62b1952: Status 404 returned error can't find the container with id b6f19d0b627f95296c6bff087b2537c52c45226e040331d8ae29055ac62b1952 Nov 02 23:25:36.471347 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:36.471306 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:25:36.471441 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:36.471406 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:25:36.471501 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:36.471451 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:25:37.364733 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:37.364695 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" event={"ID":"336646a8-fbe7-4780-85b8-116a012ea309","Type":"ContainerStarted","Data":"b76bf7f54e6edcf8a6066bf539a38bfb4acb4f8d39747ae5578f4084c022314c"} Nov 02 23:25:37.365267 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:37.364752 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" event={"ID":"336646a8-fbe7-4780-85b8-116a012ea309","Type":"ContainerStarted","Data":"b6f19d0b627f95296c6bff087b2537c52c45226e040331d8ae29055ac62b1952"} Nov 02 23:25:37.534334 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:37.534265 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" podStartSLOduration=2.534245805 podStartE2EDuration="2.534245805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:25:37.533051283 +0000 UTC m=+2212.886084658" watchObservedRunningTime="2026-11-02 23:25:37.534245805 +0000 UTC m=+2212.887279186" Nov 02 23:25:37.583925 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:25:37.583883 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:25:38.018740 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:38.018706 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:38.027340 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:38.027310 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:38.371039 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:38.371006 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:25:38.372163 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:25:38.372142 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:26:07.584565 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:26:07.584521 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:26:37.584540 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:26:37.584439 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:26:45.248319 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:26:45.248284 5114 scope.go:122] "RemoveContainer" containerID="7727144108d098a896ebf55ad26a177f6dbdc314473887ce48e3c3584f019c4b" Nov 02 23:27:07.610168 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:27:07.610123 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:27:21.130029 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.129923 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl"] Nov 02 23:27:21.134577 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.134539 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.186661 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.186628 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-818fe8-openshift-default-dockercfg-j2vfc\"" Nov 02 23:27:21.200920 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.200882 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-envoy\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.201156 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.201135 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-token\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.201330 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.201306 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-podinfo\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.201424 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.201353 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-workload-certs\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.201547 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.201521 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-credential-socket\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.201659 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.201626 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv89k\" (UniqueName: \"kubernetes.io/projected/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-kube-api-access-kv89k\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.201735 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.201694 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-workload-socket\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.201826 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.201800 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istiod-ca-cert\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.201961 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.201868 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-data\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.238836 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.238802 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl"] Nov 02 23:27:21.328522 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.303233 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-credential-socket\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.328522 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.303344 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv89k\" (UniqueName: \"kubernetes.io/projected/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-kube-api-access-kv89k\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.328522 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.303384 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-workload-socket\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.328522 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.303458 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istiod-ca-cert\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.328522 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.303518 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-data\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.328522 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.303567 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-envoy\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.328522 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.303598 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-token\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.328522 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.303722 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-podinfo\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.328522 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.303767 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-workload-certs\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.328522 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.304302 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-workload-certs\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.328522 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.304632 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-credential-socket\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.328522 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.305310 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-workload-socket\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.328522 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.309917 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-data\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.328522 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.310567 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-envoy\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.352867 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.329211 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-podinfo\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.353804 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.353711 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istiod-ca-cert\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.379513 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.379000 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv89k\" (UniqueName: \"kubernetes.io/projected/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-kube-api-access-kv89k\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.428133 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.428044 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-token\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.452059 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.452019 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:21.888444 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.888411 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl"] Nov 02 23:27:21.891590 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:27:21.891557 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc01b1c_b566_4d06_bd25_c99d8b60ac3c.slice/crio-6c93c85232a7cd71019e7e6247a52929ef63bf75ce491ac7b49045c14532e786 WatchSource:0}: Error finding container 6c93c85232a7cd71019e7e6247a52929ef63bf75ce491ac7b49045c14532e786: Status 404 returned error can't find the container with id 6c93c85232a7cd71019e7e6247a52929ef63bf75ce491ac7b49045c14532e786 Nov 02 23:27:21.894517 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.894439 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:27:21.894663 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.894579 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:27:21.894663 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.894630 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:27:21.966322 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:21.966289 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" event={"ID":"afc01b1c-b566-4d06-bd25-c99d8b60ac3c","Type":"ContainerStarted","Data":"6c93c85232a7cd71019e7e6247a52929ef63bf75ce491ac7b49045c14532e786"} Nov 02 23:27:22.973375 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:22.973334 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" event={"ID":"afc01b1c-b566-4d06-bd25-c99d8b60ac3c","Type":"ContainerStarted","Data":"1a19aca8352227c2e610d9e58d332ad83197d6f35c51312905405c7af2c7ab00"} Nov 02 23:27:23.093995 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:23.093928 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" podStartSLOduration=2.093907312 podStartE2EDuration="2.093907312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:27:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:27:23.093036073 +0000 UTC m=+2318.446069453" watchObservedRunningTime="2026-11-02 23:27:23.093907312 +0000 UTC m=+2318.446940688" Nov 02 23:27:23.453366 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:23.453318 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:23.459271 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:23.459234 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:23.979975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:23.979943 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:23.981195 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:23.981157 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:27:24.413678 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:24.413639 5114 patch_prober.go:29] interesting pod/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:27:24.420036 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:24.413705 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" podUID="336646a8-fbe7-4780-85b8-116a012ea309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:27:24.986872 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:24.986835 5114 generic.go:356] "Generic (PLEG): container finished" podID="336646a8-fbe7-4780-85b8-116a012ea309" containerID="b76bf7f54e6edcf8a6066bf539a38bfb4acb4f8d39747ae5578f4084c022314c" exitCode=0 Nov 02 23:27:24.987307 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:24.986912 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" event={"ID":"336646a8-fbe7-4780-85b8-116a012ea309","Type":"ContainerDied","Data":"b76bf7f54e6edcf8a6066bf539a38bfb4acb4f8d39747ae5578f4084c022314c"} Nov 02 23:27:24.987558 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:24.987539 5114 scope.go:122] "RemoveContainer" containerID="b76bf7f54e6edcf8a6066bf539a38bfb4acb4f8d39747ae5578f4084c022314c" Nov 02 23:27:24.989017 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:24.988978 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:27:24.989139 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:24.989073 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:27:24.989139 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:24.989122 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:27:25.994802 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:25.994764 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" event={"ID":"336646a8-fbe7-4780-85b8-116a012ea309","Type":"ContainerStarted","Data":"345384af6035acab85499fd1250cf64f75bc32512ddc8bd04197e4d57ec9b8ac"} Nov 02 23:27:26.026254 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:26.026101 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:27:26.026629 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:26.026216 5114 patch_prober.go:29] interesting pod/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 23:27:26.026838 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:26.026801 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" podUID="336646a8-fbe7-4780-85b8-116a012ea309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 23:27:27.026509 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:27.019996 5114 patch_prober.go:29] interesting pod/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 23:27:27.026509 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:27.020070 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" podUID="336646a8-fbe7-4780-85b8-116a012ea309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 23:27:28.030508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:28.029944 5114 patch_prober.go:29] interesting pod/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 23:27:28.030508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:28.030016 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" podUID="336646a8-fbe7-4780-85b8-116a012ea309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 23:27:29.026134 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:29.026094 5114 patch_prober.go:29] interesting pod/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 23:27:29.028133 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:29.028094 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" podUID="336646a8-fbe7-4780-85b8-116a012ea309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 23:27:30.025930 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:30.025886 5114 patch_prober.go:29] interesting pod/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 23:27:30.026583 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:30.026550 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" podUID="336646a8-fbe7-4780-85b8-116a012ea309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 23:27:31.023367 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:31.023319 5114 patch_prober.go:29] interesting pod/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 23:27:31.023633 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:31.023397 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" podUID="336646a8-fbe7-4780-85b8-116a012ea309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 23:27:32.023650 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.023615 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:27:32.024141 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.023965 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:27:32.024848 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.024826 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:27:32.276255 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.276176 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7"] Nov 02 23:27:32.280339 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.280316 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:27:32.337363 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.337329 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-1cb1f8\"/\"openshift-service-ca.crt\"" Nov 02 23:27:32.337363 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.337326 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-1cb1f8\"/\"kube-root-ca.crt\"" Nov 02 23:27:32.337652 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.337549 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-1cb1f8\"/\"default-dockercfg-scjsj\"" Nov 02 23:27:32.337811 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.337783 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-1cb1f8\"/\"ipp-route-1cb1f8-kserve-self-signed-certs\"" Nov 02 23:27:32.346210 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.346175 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7"] Nov 02 23:27:32.381231 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.381189 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e3d93da-6d85-4268-9261-403c456aa765-tls-certs\") pod \"ipp-route-1cb1f8-kserve-959869577-2lsd7\" (UID: \"6e3d93da-6d85-4268-9261-403c456aa765\") " pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:27:32.381410 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.381312 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-tmp-dir\") pod \"ipp-route-1cb1f8-kserve-959869577-2lsd7\" (UID: \"6e3d93da-6d85-4268-9261-403c456aa765\") " pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:27:32.381410 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.381338 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-545qz\" (UniqueName: \"kubernetes.io/projected/6e3d93da-6d85-4268-9261-403c456aa765-kube-api-access-545qz\") pod \"ipp-route-1cb1f8-kserve-959869577-2lsd7\" (UID: \"6e3d93da-6d85-4268-9261-403c456aa765\") " pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:27:32.381410 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.381403 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-dshm\") pod \"ipp-route-1cb1f8-kserve-959869577-2lsd7\" (UID: \"6e3d93da-6d85-4268-9261-403c456aa765\") " pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:27:32.381618 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.381598 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-home\") pod \"ipp-route-1cb1f8-kserve-959869577-2lsd7\" (UID: \"6e3d93da-6d85-4268-9261-403c456aa765\") " pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:27:32.381706 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.381691 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-model-cache\") pod \"ipp-route-1cb1f8-kserve-959869577-2lsd7\" (UID: \"6e3d93da-6d85-4268-9261-403c456aa765\") " pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:27:32.482337 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.482293 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-home\") pod \"ipp-route-1cb1f8-kserve-959869577-2lsd7\" (UID: \"6e3d93da-6d85-4268-9261-403c456aa765\") " pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:27:32.482563 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.482396 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-model-cache\") pod \"ipp-route-1cb1f8-kserve-959869577-2lsd7\" (UID: \"6e3d93da-6d85-4268-9261-403c456aa765\") " pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:27:32.482563 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.482436 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e3d93da-6d85-4268-9261-403c456aa765-tls-certs\") pod \"ipp-route-1cb1f8-kserve-959869577-2lsd7\" (UID: \"6e3d93da-6d85-4268-9261-403c456aa765\") " pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:27:32.482563 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.482517 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-tmp-dir\") pod \"ipp-route-1cb1f8-kserve-959869577-2lsd7\" (UID: \"6e3d93da-6d85-4268-9261-403c456aa765\") " pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:27:32.482563 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.482542 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-545qz\" (UniqueName: \"kubernetes.io/projected/6e3d93da-6d85-4268-9261-403c456aa765-kube-api-access-545qz\") pod \"ipp-route-1cb1f8-kserve-959869577-2lsd7\" (UID: \"6e3d93da-6d85-4268-9261-403c456aa765\") " pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:27:32.482788 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.482588 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-dshm\") pod \"ipp-route-1cb1f8-kserve-959869577-2lsd7\" (UID: \"6e3d93da-6d85-4268-9261-403c456aa765\") " pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:27:32.482788 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.482756 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-home\") pod \"ipp-route-1cb1f8-kserve-959869577-2lsd7\" (UID: \"6e3d93da-6d85-4268-9261-403c456aa765\") " pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:27:32.482902 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.482818 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-model-cache\") pod \"ipp-route-1cb1f8-kserve-959869577-2lsd7\" (UID: \"6e3d93da-6d85-4268-9261-403c456aa765\") " pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:27:32.483047 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.482996 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-tmp-dir\") pod \"ipp-route-1cb1f8-kserve-959869577-2lsd7\" (UID: \"6e3d93da-6d85-4268-9261-403c456aa765\") " pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:27:32.485873 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.485846 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-dshm\") pod \"ipp-route-1cb1f8-kserve-959869577-2lsd7\" (UID: \"6e3d93da-6d85-4268-9261-403c456aa765\") " pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:27:32.486069 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.486054 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e3d93da-6d85-4268-9261-403c456aa765-tls-certs\") pod \"ipp-route-1cb1f8-kserve-959869577-2lsd7\" (UID: \"6e3d93da-6d85-4268-9261-403c456aa765\") " pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:27:32.547965 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.546662 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-545qz\" (UniqueName: \"kubernetes.io/projected/6e3d93da-6d85-4268-9261-403c456aa765-kube-api-access-545qz\") pod \"ipp-route-1cb1f8-kserve-959869577-2lsd7\" (UID: \"6e3d93da-6d85-4268-9261-403c456aa765\") " pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:27:32.595014 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.594970 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:27:32.988526 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:32.988497 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7"] Nov 02 23:27:32.990902 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:27:32.990868 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e3d93da_6d85_4268_9261_403c456aa765.slice/crio-b3d4752989cdc6981461d11d09a1c357c6375f220d50d9ccda26a7c8813b3a29 WatchSource:0}: Error finding container b3d4752989cdc6981461d11d09a1c357c6375f220d50d9ccda26a7c8813b3a29: Status 404 returned error can't find the container with id b3d4752989cdc6981461d11d09a1c357c6375f220d50d9ccda26a7c8813b3a29 Nov 02 23:27:33.056778 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:33.056733 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" event={"ID":"6e3d93da-6d85-4268-9261-403c456aa765","Type":"ContainerStarted","Data":"55fc1d02d62a701eca53449e1b27d85bad933d7029f2448cdd0e8ab0e7d9f78b"} Nov 02 23:27:33.057196 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:33.056792 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" event={"ID":"6e3d93da-6d85-4268-9261-403c456aa765","Type":"ContainerStarted","Data":"b3d4752989cdc6981461d11d09a1c357c6375f220d50d9ccda26a7c8813b3a29"} Nov 02 23:27:33.174880 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:33.174817 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" podStartSLOduration=1.174799438 podStartE2EDuration="1.174799438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:27:33.173102392 +0000 UTC m=+2328.526135764" watchObservedRunningTime="2026-11-02 23:27:33.174799438 +0000 UTC m=+2328.527832822" Nov 02 23:27:33.595155 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:33.595113 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:27:37.584152 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:27:37.584107 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:27:42.596059 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:42.596013 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:27:42.609645 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:42.609611 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:27:43.134439 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:27:43.134409 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:28:07.585035 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:28:07.584987 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:28:26.216445 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:26.216403 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld"] Nov 02 23:28:26.217148 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:26.216747 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" podUID="1c9af20a-578a-4416-aee4-5fbdcb3370c4" containerName="istio-proxy" containerID="cri-o://6248671ddcbd73b3791df7be705f0a7cdc98d4deb1246ce5c5d9e35e1ca193ce" gracePeriod=30 Nov 02 23:28:26.331206 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:26.331171 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws"] Nov 02 23:28:26.331418 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:26.331387 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" podUID="774684c7-0c28-43c7-bba7-d2c1bcccad16" containerName="istio-proxy" containerID="cri-o://a1dff3951ca35c7200893e7aff7631d395972abde1cf7d99c47aeb4ec0df36bd" gracePeriod=30 Nov 02 23:28:30.396733 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:30.396692 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:28:30.397169 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:30.396766 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" podUID="1c9af20a-578a-4416-aee4-5fbdcb3370c4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:28:31.388369 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.388334 5114 generic.go:356] "Generic (PLEG): container finished" podID="1c9af20a-578a-4416-aee4-5fbdcb3370c4" containerID="6248671ddcbd73b3791df7be705f0a7cdc98d4deb1246ce5c5d9e35e1ca193ce" exitCode=0 Nov 02 23:28:31.388571 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.388405 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" event={"ID":"1c9af20a-578a-4416-aee4-5fbdcb3370c4","Type":"ContainerDied","Data":"6248671ddcbd73b3791df7be705f0a7cdc98d4deb1246ce5c5d9e35e1ca193ce"} Nov 02 23:28:31.492670 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.492597 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:28:31.604204 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.604168 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:28:31.649788 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.649748 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-podinfo\") pod \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " Nov 02 23:28:31.650001 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.649819 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c9af20a-578a-4416-aee4-5fbdcb3370c4-kube-api-access-vlmpq\" (UniqueName: \"kubernetes.io/projected/1c9af20a-578a-4416-aee4-5fbdcb3370c4-kube-api-access-vlmpq\") pod \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " Nov 02 23:28:31.650001 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.649855 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-token\" (UniqueName: \"kubernetes.io/projected/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-token\") pod \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " Nov 02 23:28:31.650001 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.649932 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-credential-socket\") pod \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " Nov 02 23:28:31.650001 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.649950 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istiod-ca-cert\") pod \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " Nov 02 23:28:31.650239 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.650004 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-workload-certs\") pod \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " Nov 02 23:28:31.650239 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.650058 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-workload-socket\") pod \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " Nov 02 23:28:31.650239 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.650105 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-data\") pod \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " Nov 02 23:28:31.650239 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.650190 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-envoy\") pod \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\" (UID: \"1c9af20a-578a-4416-aee4-5fbdcb3370c4\") " Nov 02 23:28:31.650239 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.650221 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-credential-socket" pod "1c9af20a-578a-4416-aee4-5fbdcb3370c4" (UID: "1c9af20a-578a-4416-aee4-5fbdcb3370c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:31.650539 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.650377 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-workload-socket" pod "1c9af20a-578a-4416-aee4-5fbdcb3370c4" (UID: "1c9af20a-578a-4416-aee4-5fbdcb3370c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:31.650539 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.650447 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istiod-ca-cert" pod "1c9af20a-578a-4416-aee4-5fbdcb3370c4" (UID: "1c9af20a-578a-4416-aee4-5fbdcb3370c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:28:31.650539 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.650441 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-data" pod "1c9af20a-578a-4416-aee4-5fbdcb3370c4" (UID: "1c9af20a-578a-4416-aee4-5fbdcb3370c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:31.650864 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.650554 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-workload-certs" pod "1c9af20a-578a-4416-aee4-5fbdcb3370c4" (UID: "1c9af20a-578a-4416-aee4-5fbdcb3370c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:31.650864 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.650754 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:31.650864 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.650770 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:31.650864 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.650779 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:31.650864 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.650787 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:31.650864 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.650795 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:31.652388 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.652364 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9af20a-578a-4416-aee4-5fbdcb3370c4-kube-api-access-vlmpq" pod "1c9af20a-578a-4416-aee4-5fbdcb3370c4" (UID: "1c9af20a-578a-4416-aee4-5fbdcb3370c4"). InnerVolumeSpecName "kube-api-access-vlmpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:31.652541 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.652447 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-envoy" pod "1c9af20a-578a-4416-aee4-5fbdcb3370c4" (UID: "1c9af20a-578a-4416-aee4-5fbdcb3370c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:31.652789 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.652773 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-podinfo" pod "1c9af20a-578a-4416-aee4-5fbdcb3370c4" (UID: "1c9af20a-578a-4416-aee4-5fbdcb3370c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:28:31.652863 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.652809 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-token" pod "1c9af20a-578a-4416-aee4-5fbdcb3370c4" (UID: "1c9af20a-578a-4416-aee4-5fbdcb3370c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:31.751282 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.751196 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-credential-socket\") pod \"774684c7-0c28-43c7-bba7-d2c1bcccad16\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " Nov 02 23:28:31.751282 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.751266 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-data\") pod \"774684c7-0c28-43c7-bba7-d2c1bcccad16\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " Nov 02 23:28:31.751548 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.751289 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-envoy\") pod \"774684c7-0c28-43c7-bba7-d2c1bcccad16\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " Nov 02 23:28:31.751548 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.751336 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-workload-socket\") pod \"774684c7-0c28-43c7-bba7-d2c1bcccad16\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " Nov 02 23:28:31.751548 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.751368 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-workload-certs\") pod \"774684c7-0c28-43c7-bba7-d2c1bcccad16\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " Nov 02 23:28:31.751548 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.751439 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/774684c7-0c28-43c7-bba7-d2c1bcccad16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774684c7-0c28-43c7-bba7-d2c1bcccad16-istiod-ca-cert\") pod \"774684c7-0c28-43c7-bba7-d2c1bcccad16\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " Nov 02 23:28:31.751548 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.751514 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-podinfo\") pod \"774684c7-0c28-43c7-bba7-d2c1bcccad16\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " Nov 02 23:28:31.751819 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.751554 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774684c7-0c28-43c7-bba7-d2c1bcccad16-kube-api-access-2pchb\" (UniqueName: \"kubernetes.io/projected/774684c7-0c28-43c7-bba7-d2c1bcccad16-kube-api-access-2pchb\") pod \"774684c7-0c28-43c7-bba7-d2c1bcccad16\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " Nov 02 23:28:31.751819 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.751571 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-data" pod "774684c7-0c28-43c7-bba7-d2c1bcccad16" (UID: "774684c7-0c28-43c7-bba7-d2c1bcccad16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:31.751819 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.751579 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-credential-socket" pod "774684c7-0c28-43c7-bba7-d2c1bcccad16" (UID: "774684c7-0c28-43c7-bba7-d2c1bcccad16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:31.751819 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.751598 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-token\" (UniqueName: \"kubernetes.io/projected/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-token\") pod \"774684c7-0c28-43c7-bba7-d2c1bcccad16\" (UID: \"774684c7-0c28-43c7-bba7-d2c1bcccad16\") " Nov 02 23:28:31.751819 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.751703 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-workload-socket" pod "774684c7-0c28-43c7-bba7-d2c1bcccad16" (UID: "774684c7-0c28-43c7-bba7-d2c1bcccad16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:31.752076 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.751842 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-workload-certs" pod "774684c7-0c28-43c7-bba7-d2c1bcccad16" (UID: "774684c7-0c28-43c7-bba7-d2c1bcccad16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:31.752076 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.752022 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/774684c7-0c28-43c7-bba7-d2c1bcccad16-istiod-ca-cert" pod "774684c7-0c28-43c7-bba7-d2c1bcccad16" (UID: "774684c7-0c28-43c7-bba7-d2c1bcccad16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:28:31.752351 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.752093 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlmpq\" (UniqueName: \"kubernetes.io/projected/1c9af20a-578a-4416-aee4-5fbdcb3370c4-kube-api-access-vlmpq\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:31.752351 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.752116 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:31.752351 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.752132 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:31.752351 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.752146 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:31.752351 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.752160 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:31.752351 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.752173 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:31.752351 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.752186 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c9af20a-578a-4416-aee4-5fbdcb3370c4-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:31.752351 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.752199 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:31.754030 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.753994 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-podinfo" pod "774684c7-0c28-43c7-bba7-d2c1bcccad16" (UID: "774684c7-0c28-43c7-bba7-d2c1bcccad16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:28:31.754154 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.754024 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-envoy" pod "774684c7-0c28-43c7-bba7-d2c1bcccad16" (UID: "774684c7-0c28-43c7-bba7-d2c1bcccad16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:31.754154 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.754047 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774684c7-0c28-43c7-bba7-d2c1bcccad16-kube-api-access-2pchb" pod "774684c7-0c28-43c7-bba7-d2c1bcccad16" (UID: "774684c7-0c28-43c7-bba7-d2c1bcccad16"). InnerVolumeSpecName "kube-api-access-2pchb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:31.754154 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.754060 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-token" pod "774684c7-0c28-43c7-bba7-d2c1bcccad16" (UID: "774684c7-0c28-43c7-bba7-d2c1bcccad16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:31.853430 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.853391 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774684c7-0c28-43c7-bba7-d2c1bcccad16-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:31.853430 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.853427 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:31.853430 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.853436 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pchb\" (UniqueName: \"kubernetes.io/projected/774684c7-0c28-43c7-bba7-d2c1bcccad16-kube-api-access-2pchb\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:31.853430 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.853448 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:31.853739 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:31.853458 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774684c7-0c28-43c7-bba7-d2c1bcccad16-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:32.395452 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:32.395418 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" Nov 02 23:28:32.395680 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:32.395417 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld" event={"ID":"1c9af20a-578a-4416-aee4-5fbdcb3370c4","Type":"ContainerDied","Data":"7d6d097d605b8f5b4d46be70602c2e7ff4fdffe634961343ea31276143a24d0a"} Nov 02 23:28:32.395680 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:32.395555 5114 scope.go:122] "RemoveContainer" containerID="6248671ddcbd73b3791df7be705f0a7cdc98d4deb1246ce5c5d9e35e1ca193ce" Nov 02 23:28:32.397065 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:32.397042 5114 generic.go:356] "Generic (PLEG): container finished" podID="774684c7-0c28-43c7-bba7-d2c1bcccad16" containerID="a1dff3951ca35c7200893e7aff7631d395972abde1cf7d99c47aeb4ec0df36bd" exitCode=0 Nov 02 23:28:32.397165 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:32.397115 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" Nov 02 23:28:32.397165 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:32.397127 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" event={"ID":"774684c7-0c28-43c7-bba7-d2c1bcccad16","Type":"ContainerDied","Data":"a1dff3951ca35c7200893e7aff7631d395972abde1cf7d99c47aeb4ec0df36bd"} Nov 02 23:28:32.397271 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:32.397188 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws" event={"ID":"774684c7-0c28-43c7-bba7-d2c1bcccad16","Type":"ContainerDied","Data":"1fa365df914a7da1d274511b102567ea242481fbe06faad4a44b9818c2dd9d71"} Nov 02 23:28:32.405909 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:32.405884 5114 scope.go:122] "RemoveContainer" containerID="a1dff3951ca35c7200893e7aff7631d395972abde1cf7d99c47aeb4ec0df36bd" Nov 02 23:28:32.415284 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:32.415258 5114 scope.go:122] "RemoveContainer" containerID="a1dff3951ca35c7200893e7aff7631d395972abde1cf7d99c47aeb4ec0df36bd" Nov 02 23:28:32.415704 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:28:32.415675 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1dff3951ca35c7200893e7aff7631d395972abde1cf7d99c47aeb4ec0df36bd\": container with ID starting with a1dff3951ca35c7200893e7aff7631d395972abde1cf7d99c47aeb4ec0df36bd not found: ID does not exist" containerID="a1dff3951ca35c7200893e7aff7631d395972abde1cf7d99c47aeb4ec0df36bd" Nov 02 23:28:32.415797 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:32.415714 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1dff3951ca35c7200893e7aff7631d395972abde1cf7d99c47aeb4ec0df36bd"} err="failed to get container status \"a1dff3951ca35c7200893e7aff7631d395972abde1cf7d99c47aeb4ec0df36bd\": rpc error: code = NotFound desc = could not find container \"a1dff3951ca35c7200893e7aff7631d395972abde1cf7d99c47aeb4ec0df36bd\": container with ID starting with a1dff3951ca35c7200893e7aff7631d395972abde1cf7d99c47aeb4ec0df36bd not found: ID does not exist" Nov 02 23:28:32.617515 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:32.617464 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws"] Nov 02 23:28:32.622190 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:32.622153 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-btmws"] Nov 02 23:28:32.731529 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:32.731473 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld"] Nov 02 23:28:32.734582 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:32.734553 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-6h6ld"] Nov 02 23:28:32.797185 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:32.797129 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c9af20a-578a-4416-aee4-5fbdcb3370c4" path="/var/lib/kubelet/pods/1c9af20a-578a-4416-aee4-5fbdcb3370c4/volumes" Nov 02 23:28:32.797991 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:32.797968 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="774684c7-0c28-43c7-bba7-d2c1bcccad16" path="/var/lib/kubelet/pods/774684c7-0c28-43c7-bba7-d2c1bcccad16/volumes" Nov 02 23:28:32.919098 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:32.918997 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf"] Nov 02 23:28:32.919434 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:32.919375 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" podUID="afcd99fa-b983-4c6d-83a6-43cd089a8f5e" containerName="istio-proxy" containerID="cri-o://083de0e1b597425c6322de1b65f231e9af6f642b6efc2a64ac0115d0f8443f60" gracePeriod=30 Nov 02 23:28:35.351883 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:35.351845 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf"] Nov 02 23:28:35.352334 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:35.352065 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" podUID="6aa4c136-742e-4387-b751-87692a034ba2" containerName="istio-proxy" containerID="cri-o://41634aae1ce9ce1a01cabfeb30ea5867244fbb8c0e5a004a891285754b1d7cd8" gracePeriod=30 Nov 02 23:28:35.559975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:35.559941 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-jgsxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:28:35.560143 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:35.560012 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" podUID="6aa4c136-742e-4387-b751-87692a034ba2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:28:37.584350 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:28:37.584299 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:28:38.230248 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.230223 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:28:38.330404 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.330361 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-data\") pod \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " Nov 02 23:28:38.330597 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.330432 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istiod-ca-cert\") pod \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " Nov 02 23:28:38.330597 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.330545 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-token\" (UniqueName: \"kubernetes.io/projected/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-token\") pod \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " Nov 02 23:28:38.330597 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.330574 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-kube-api-access-955v8\" (UniqueName: \"kubernetes.io/projected/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-kube-api-access-955v8\") pod \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " Nov 02 23:28:38.330791 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.330607 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-envoy\") pod \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " Nov 02 23:28:38.330791 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.330629 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-workload-socket\") pod \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " Nov 02 23:28:38.330791 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.330679 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-podinfo\") pod \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " Nov 02 23:28:38.330791 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.330745 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-credential-socket\") pod \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " Nov 02 23:28:38.330791 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.330758 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-data" pod "afcd99fa-b983-4c6d-83a6-43cd089a8f5e" (UID: "afcd99fa-b983-4c6d-83a6-43cd089a8f5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:38.331045 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.330831 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-workload-certs\") pod \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\" (UID: \"afcd99fa-b983-4c6d-83a6-43cd089a8f5e\") " Nov 02 23:28:38.331045 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.330877 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istiod-ca-cert" pod "afcd99fa-b983-4c6d-83a6-43cd089a8f5e" (UID: "afcd99fa-b983-4c6d-83a6-43cd089a8f5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:28:38.331147 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.331025 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-workload-socket" pod "afcd99fa-b983-4c6d-83a6-43cd089a8f5e" (UID: "afcd99fa-b983-4c6d-83a6-43cd089a8f5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:38.331241 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.331195 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-credential-socket" pod "afcd99fa-b983-4c6d-83a6-43cd089a8f5e" (UID: "afcd99fa-b983-4c6d-83a6-43cd089a8f5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:38.331241 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.331228 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-workload-certs" pod "afcd99fa-b983-4c6d-83a6-43cd089a8f5e" (UID: "afcd99fa-b983-4c6d-83a6-43cd089a8f5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:38.331726 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.331621 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:38.331726 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.331644 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:38.331726 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.331659 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:38.331726 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.331673 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:38.331726 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.331688 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:38.333168 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.333141 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-envoy" pod "afcd99fa-b983-4c6d-83a6-43cd089a8f5e" (UID: "afcd99fa-b983-4c6d-83a6-43cd089a8f5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:38.333274 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.333184 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-kube-api-access-955v8" pod "afcd99fa-b983-4c6d-83a6-43cd089a8f5e" (UID: "afcd99fa-b983-4c6d-83a6-43cd089a8f5e"). InnerVolumeSpecName "kube-api-access-955v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:38.333274 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.333214 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-podinfo" pod "afcd99fa-b983-4c6d-83a6-43cd089a8f5e" (UID: "afcd99fa-b983-4c6d-83a6-43cd089a8f5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:28:38.333437 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.333423 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-token" pod "afcd99fa-b983-4c6d-83a6-43cd089a8f5e" (UID: "afcd99fa-b983-4c6d-83a6-43cd089a8f5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:38.432882 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.432840 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:38.432882 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.432877 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-955v8\" (UniqueName: \"kubernetes.io/projected/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-kube-api-access-955v8\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:38.432882 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.432887 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:38.433141 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.432896 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afcd99fa-b983-4c6d-83a6-43cd089a8f5e-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:38.444988 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.444948 5114 generic.go:356] "Generic (PLEG): container finished" podID="afcd99fa-b983-4c6d-83a6-43cd089a8f5e" containerID="083de0e1b597425c6322de1b65f231e9af6f642b6efc2a64ac0115d0f8443f60" exitCode=0 Nov 02 23:28:38.445201 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.445015 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" Nov 02 23:28:38.445201 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.445030 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" event={"ID":"afcd99fa-b983-4c6d-83a6-43cd089a8f5e","Type":"ContainerDied","Data":"083de0e1b597425c6322de1b65f231e9af6f642b6efc2a64ac0115d0f8443f60"} Nov 02 23:28:38.445201 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.445098 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf" event={"ID":"afcd99fa-b983-4c6d-83a6-43cd089a8f5e","Type":"ContainerDied","Data":"d31a6136abd0f0d1ee81ca2aff907fc9d923527f27a1d4d1a9bae7621e49e54d"} Nov 02 23:28:38.445201 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.445127 5114 scope.go:122] "RemoveContainer" containerID="083de0e1b597425c6322de1b65f231e9af6f642b6efc2a64ac0115d0f8443f60" Nov 02 23:28:38.455465 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.455443 5114 scope.go:122] "RemoveContainer" containerID="083de0e1b597425c6322de1b65f231e9af6f642b6efc2a64ac0115d0f8443f60" Nov 02 23:28:38.455824 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:28:38.455801 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"083de0e1b597425c6322de1b65f231e9af6f642b6efc2a64ac0115d0f8443f60\": container with ID starting with 083de0e1b597425c6322de1b65f231e9af6f642b6efc2a64ac0115d0f8443f60 not found: ID does not exist" containerID="083de0e1b597425c6322de1b65f231e9af6f642b6efc2a64ac0115d0f8443f60" Nov 02 23:28:38.455886 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.455834 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"083de0e1b597425c6322de1b65f231e9af6f642b6efc2a64ac0115d0f8443f60"} err="failed to get container status \"083de0e1b597425c6322de1b65f231e9af6f642b6efc2a64ac0115d0f8443f60\": rpc error: code = NotFound desc = could not find container \"083de0e1b597425c6322de1b65f231e9af6f642b6efc2a64ac0115d0f8443f60\": container with ID starting with 083de0e1b597425c6322de1b65f231e9af6f642b6efc2a64ac0115d0f8443f60 not found: ID does not exist" Nov 02 23:28:38.609822 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.609786 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf"] Nov 02 23:28:38.615512 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.615464 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-b4rsf"] Nov 02 23:28:38.794130 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:38.794088 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afcd99fa-b983-4c6d-83a6-43cd089a8f5e" path="/var/lib/kubelet/pods/afcd99fa-b983-4c6d-83a6-43cd089a8f5e/volumes" Nov 02 23:28:40.624281 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.624212 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 23:28:40.759791 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.759756 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-istio-envoy\") pod \"6aa4c136-742e-4387-b751-87692a034ba2\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " Nov 02 23:28:40.759968 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.759799 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-workload-socket\") pod \"6aa4c136-742e-4387-b751-87692a034ba2\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " Nov 02 23:28:40.759968 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.759820 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa4c136-742e-4387-b751-87692a034ba2-kube-api-access-bfqtg\" (UniqueName: \"kubernetes.io/projected/6aa4c136-742e-4387-b751-87692a034ba2-kube-api-access-bfqtg\") pod \"6aa4c136-742e-4387-b751-87692a034ba2\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " Nov 02 23:28:40.759968 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.759852 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-istio-data\") pod \"6aa4c136-742e-4387-b751-87692a034ba2\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " Nov 02 23:28:40.759968 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.759922 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aa4c136-742e-4387-b751-87692a034ba2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa4c136-742e-4387-b751-87692a034ba2-istiod-ca-cert\") pod \"6aa4c136-742e-4387-b751-87692a034ba2\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " Nov 02 23:28:40.760205 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.760012 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6aa4c136-742e-4387-b751-87692a034ba2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa4c136-742e-4387-b751-87692a034ba2-istio-podinfo\") pod \"6aa4c136-742e-4387-b751-87692a034ba2\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " Nov 02 23:28:40.760205 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.760058 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-credential-socket\") pod \"6aa4c136-742e-4387-b751-87692a034ba2\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " Nov 02 23:28:40.760205 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.760090 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-workload-socket" pod "6aa4c136-742e-4387-b751-87692a034ba2" (UID: "6aa4c136-742e-4387-b751-87692a034ba2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:40.760205 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.760116 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa4c136-742e-4387-b751-87692a034ba2-istio-token\" (UniqueName: \"kubernetes.io/projected/6aa4c136-742e-4387-b751-87692a034ba2-istio-token\") pod \"6aa4c136-742e-4387-b751-87692a034ba2\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " Nov 02 23:28:40.760205 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.760157 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-workload-certs\") pod \"6aa4c136-742e-4387-b751-87692a034ba2\" (UID: \"6aa4c136-742e-4387-b751-87692a034ba2\") " Nov 02 23:28:40.760461 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.760267 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-credential-socket" pod "6aa4c136-742e-4387-b751-87692a034ba2" (UID: "6aa4c136-742e-4387-b751-87692a034ba2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:40.760461 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.760399 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa4c136-742e-4387-b751-87692a034ba2-istiod-ca-cert" pod "6aa4c136-742e-4387-b751-87692a034ba2" (UID: "6aa4c136-742e-4387-b751-87692a034ba2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:28:40.760654 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.760617 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-workload-certs" pod "6aa4c136-742e-4387-b751-87692a034ba2" (UID: "6aa4c136-742e-4387-b751-87692a034ba2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:40.760794 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.760652 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-istio-data" pod "6aa4c136-742e-4387-b751-87692a034ba2" (UID: "6aa4c136-742e-4387-b751-87692a034ba2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:40.761163 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.761140 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:40.761225 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.761169 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:40.761225 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.761185 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa4c136-742e-4387-b751-87692a034ba2-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:40.761225 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.761200 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:40.761225 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.761216 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:40.762352 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.762330 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa4c136-742e-4387-b751-87692a034ba2-kube-api-access-bfqtg" pod "6aa4c136-742e-4387-b751-87692a034ba2" (UID: "6aa4c136-742e-4387-b751-87692a034ba2"). InnerVolumeSpecName "kube-api-access-bfqtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:40.762582 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.762563 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa4c136-742e-4387-b751-87692a034ba2-istio-token" pod "6aa4c136-742e-4387-b751-87692a034ba2" (UID: "6aa4c136-742e-4387-b751-87692a034ba2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:40.762834 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.762811 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6aa4c136-742e-4387-b751-87692a034ba2-istio-podinfo" pod "6aa4c136-742e-4387-b751-87692a034ba2" (UID: "6aa4c136-742e-4387-b751-87692a034ba2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:28:40.762929 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.762902 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-istio-envoy" pod "6aa4c136-742e-4387-b751-87692a034ba2" (UID: "6aa4c136-742e-4387-b751-87692a034ba2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:40.862202 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.862160 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa4c136-742e-4387-b751-87692a034ba2-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:40.862202 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.862194 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa4c136-742e-4387-b751-87692a034ba2-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:40.862202 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.862204 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfqtg\" (UniqueName: \"kubernetes.io/projected/6aa4c136-742e-4387-b751-87692a034ba2-kube-api-access-bfqtg\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:40.862447 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:40.862214 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa4c136-742e-4387-b751-87692a034ba2-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:41.322167 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:41.322129 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9"] Nov 02 23:28:41.322701 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:41.322649 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" podUID="57299a9e-ed21-49ee-acdd-81831c2efd84" containerName="istio-proxy" containerID="cri-o://a47c38c0fa1246544943ebf531ba37b152705fc0c5c57950be5b80aa495b1aac" gracePeriod=30 Nov 02 23:28:41.464667 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:41.464629 5114 generic.go:356] "Generic (PLEG): container finished" podID="6aa4c136-742e-4387-b751-87692a034ba2" containerID="41634aae1ce9ce1a01cabfeb30ea5867244fbb8c0e5a004a891285754b1d7cd8" exitCode=0 Nov 02 23:28:41.464872 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:41.464697 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" Nov 02 23:28:41.464872 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:41.464702 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" event={"ID":"6aa4c136-742e-4387-b751-87692a034ba2","Type":"ContainerDied","Data":"41634aae1ce9ce1a01cabfeb30ea5867244fbb8c0e5a004a891285754b1d7cd8"} Nov 02 23:28:41.464872 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:41.464760 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf" event={"ID":"6aa4c136-742e-4387-b751-87692a034ba2","Type":"ContainerDied","Data":"ce0bc5b41e623b5b97ccc0b8f36a145574ca7f2492b81bed9e563c1adff3c20f"} Nov 02 23:28:41.464872 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:41.464782 5114 scope.go:122] "RemoveContainer" containerID="41634aae1ce9ce1a01cabfeb30ea5867244fbb8c0e5a004a891285754b1d7cd8" Nov 02 23:28:41.475213 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:41.475069 5114 scope.go:122] "RemoveContainer" containerID="41634aae1ce9ce1a01cabfeb30ea5867244fbb8c0e5a004a891285754b1d7cd8" Nov 02 23:28:41.475390 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:28:41.475362 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41634aae1ce9ce1a01cabfeb30ea5867244fbb8c0e5a004a891285754b1d7cd8\": container with ID starting with 41634aae1ce9ce1a01cabfeb30ea5867244fbb8c0e5a004a891285754b1d7cd8 not found: ID does not exist" containerID="41634aae1ce9ce1a01cabfeb30ea5867244fbb8c0e5a004a891285754b1d7cd8" Nov 02 23:28:41.475427 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:41.475402 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41634aae1ce9ce1a01cabfeb30ea5867244fbb8c0e5a004a891285754b1d7cd8"} err="failed to get container status \"41634aae1ce9ce1a01cabfeb30ea5867244fbb8c0e5a004a891285754b1d7cd8\": rpc error: code = NotFound desc = could not find container \"41634aae1ce9ce1a01cabfeb30ea5867244fbb8c0e5a004a891285754b1d7cd8\": container with ID starting with 41634aae1ce9ce1a01cabfeb30ea5867244fbb8c0e5a004a891285754b1d7cd8 not found: ID does not exist" Nov 02 23:28:41.629651 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:41.629554 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf"] Nov 02 23:28:41.633108 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:41.633073 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-jgsxf"] Nov 02 23:28:42.793426 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:42.793391 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aa4c136-742e-4387-b751-87692a034ba2" path="/var/lib/kubelet/pods/6aa4c136-742e-4387-b751-87692a034ba2/volumes" Nov 02 23:28:46.493588 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.493546 5114 generic.go:356] "Generic (PLEG): container finished" podID="57299a9e-ed21-49ee-acdd-81831c2efd84" containerID="a47c38c0fa1246544943ebf531ba37b152705fc0c5c57950be5b80aa495b1aac" exitCode=0 Nov 02 23:28:46.494049 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.493619 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" event={"ID":"57299a9e-ed21-49ee-acdd-81831c2efd84","Type":"ContainerDied","Data":"a47c38c0fa1246544943ebf531ba37b152705fc0c5c57950be5b80aa495b1aac"} Nov 02 23:28:46.494049 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.493703 5114 scope.go:122] "RemoveContainer" containerID="4aaca3c96642a52203751a35e36c120e36fcb7f70ecee0cbd6cc4e9433c4a40b" Nov 02 23:28:46.590499 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.590459 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:28:46.615472 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.615378 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-podinfo\") pod \"57299a9e-ed21-49ee-acdd-81831c2efd84\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " Nov 02 23:28:46.615752 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.615528 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57299a9e-ed21-49ee-acdd-81831c2efd84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57299a9e-ed21-49ee-acdd-81831c2efd84-istiod-ca-cert\") pod \"57299a9e-ed21-49ee-acdd-81831c2efd84\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " Nov 02 23:28:46.615752 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.615585 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-credential-socket\") pod \"57299a9e-ed21-49ee-acdd-81831c2efd84\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " Nov 02 23:28:46.615752 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.615628 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-data\") pod \"57299a9e-ed21-49ee-acdd-81831c2efd84\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " Nov 02 23:28:46.615752 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.615685 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-token\" (UniqueName: \"kubernetes.io/projected/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-token\") pod \"57299a9e-ed21-49ee-acdd-81831c2efd84\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " Nov 02 23:28:46.615752 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.615730 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-workload-socket\") pod \"57299a9e-ed21-49ee-acdd-81831c2efd84\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " Nov 02 23:28:46.616044 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.615768 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57299a9e-ed21-49ee-acdd-81831c2efd84-kube-api-access-7jxbb\" (UniqueName: \"kubernetes.io/projected/57299a9e-ed21-49ee-acdd-81831c2efd84-kube-api-access-7jxbb\") pod \"57299a9e-ed21-49ee-acdd-81831c2efd84\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " Nov 02 23:28:46.616044 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.615812 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-workload-certs\") pod \"57299a9e-ed21-49ee-acdd-81831c2efd84\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " Nov 02 23:28:46.616044 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.615850 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-envoy\") pod \"57299a9e-ed21-49ee-acdd-81831c2efd84\" (UID: \"57299a9e-ed21-49ee-acdd-81831c2efd84\") " Nov 02 23:28:46.616044 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.615869 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-credential-socket" pod "57299a9e-ed21-49ee-acdd-81831c2efd84" (UID: "57299a9e-ed21-49ee-acdd-81831c2efd84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:46.616044 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.615982 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57299a9e-ed21-49ee-acdd-81831c2efd84-istiod-ca-cert" pod "57299a9e-ed21-49ee-acdd-81831c2efd84" (UID: "57299a9e-ed21-49ee-acdd-81831c2efd84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:28:46.616279 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.616070 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-data" pod "57299a9e-ed21-49ee-acdd-81831c2efd84" (UID: "57299a9e-ed21-49ee-acdd-81831c2efd84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:46.616279 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.616184 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-workload-socket" pod "57299a9e-ed21-49ee-acdd-81831c2efd84" (UID: "57299a9e-ed21-49ee-acdd-81831c2efd84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:46.616413 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.616392 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-workload-certs" pod "57299a9e-ed21-49ee-acdd-81831c2efd84" (UID: "57299a9e-ed21-49ee-acdd-81831c2efd84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:46.616590 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.616542 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57299a9e-ed21-49ee-acdd-81831c2efd84-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:46.616808 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.616610 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:46.616808 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.616769 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:46.616808 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.616787 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:46.616808 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.616801 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:46.618264 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.618234 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-podinfo" pod "57299a9e-ed21-49ee-acdd-81831c2efd84" (UID: "57299a9e-ed21-49ee-acdd-81831c2efd84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:28:46.618264 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.618237 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57299a9e-ed21-49ee-acdd-81831c2efd84-kube-api-access-7jxbb" pod "57299a9e-ed21-49ee-acdd-81831c2efd84" (UID: "57299a9e-ed21-49ee-acdd-81831c2efd84"). InnerVolumeSpecName "kube-api-access-7jxbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:46.618439 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.618305 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-token" pod "57299a9e-ed21-49ee-acdd-81831c2efd84" (UID: "57299a9e-ed21-49ee-acdd-81831c2efd84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:46.618439 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.618362 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-envoy" pod "57299a9e-ed21-49ee-acdd-81831c2efd84" (UID: "57299a9e-ed21-49ee-acdd-81831c2efd84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:46.717887 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.717841 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:46.717887 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.717880 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jxbb\" (UniqueName: \"kubernetes.io/projected/57299a9e-ed21-49ee-acdd-81831c2efd84-kube-api-access-7jxbb\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:46.717887 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.717892 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:46.718140 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:46.717901 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57299a9e-ed21-49ee-acdd-81831c2efd84-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:47.500369 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:47.500338 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" Nov 02 23:28:47.500917 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:47.500382 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9" event={"ID":"57299a9e-ed21-49ee-acdd-81831c2efd84","Type":"ContainerDied","Data":"9b8456b27b571219b6e1e6cd8ea2b4550c69ee2f3d7db4e2bfb74c10930f6876"} Nov 02 23:28:47.500917 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:47.500445 5114 scope.go:122] "RemoveContainer" containerID="a47c38c0fa1246544943ebf531ba37b152705fc0c5c57950be5b80aa495b1aac" Nov 02 23:28:47.667688 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:47.667650 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9"] Nov 02 23:28:47.716822 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:47.716781 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-8hvw9"] Nov 02 23:28:48.793051 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:48.793016 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57299a9e-ed21-49ee-acdd-81831c2efd84" path="/var/lib/kubelet/pods/57299a9e-ed21-49ee-acdd-81831c2efd84/volumes" Nov 02 23:28:57.106043 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.106004 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj"] Nov 02 23:28:57.106596 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.106427 5114 cpu_manager.go:414] "containerMap: removing container" podUID="1c9af20a-578a-4416-aee4-5fbdcb3370c4" containerName="istio-proxy" Nov 02 23:28:57.106596 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.106439 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c9af20a-578a-4416-aee4-5fbdcb3370c4" containerName="istio-proxy" Nov 02 23:28:57.106596 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.106454 5114 cpu_manager.go:414] "containerMap: removing container" podUID="57299a9e-ed21-49ee-acdd-81831c2efd84" containerName="istio-proxy" Nov 02 23:28:57.106596 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.106459 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57299a9e-ed21-49ee-acdd-81831c2efd84" containerName="istio-proxy" Nov 02 23:28:57.106596 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.106472 5114 cpu_manager.go:414] "containerMap: removing container" podUID="6aa4c136-742e-4387-b751-87692a034ba2" containerName="istio-proxy" Nov 02 23:28:57.106596 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.106495 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa4c136-742e-4387-b751-87692a034ba2" containerName="istio-proxy" Nov 02 23:28:57.106596 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.106508 5114 cpu_manager.go:414] "containerMap: removing container" podUID="57299a9e-ed21-49ee-acdd-81831c2efd84" containerName="istio-proxy" Nov 02 23:28:57.106596 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.106515 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57299a9e-ed21-49ee-acdd-81831c2efd84" containerName="istio-proxy" Nov 02 23:28:57.106596 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.106529 5114 cpu_manager.go:414] "containerMap: removing container" podUID="afcd99fa-b983-4c6d-83a6-43cd089a8f5e" containerName="istio-proxy" Nov 02 23:28:57.106596 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.106533 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afcd99fa-b983-4c6d-83a6-43cd089a8f5e" containerName="istio-proxy" Nov 02 23:28:57.106596 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.106540 5114 cpu_manager.go:414] "containerMap: removing container" podUID="774684c7-0c28-43c7-bba7-d2c1bcccad16" containerName="istio-proxy" Nov 02 23:28:57.106596 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.106545 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="774684c7-0c28-43c7-bba7-d2c1bcccad16" containerName="istio-proxy" Nov 02 23:28:57.106968 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.106625 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c9af20a-578a-4416-aee4-5fbdcb3370c4" containerName="istio-proxy" Nov 02 23:28:57.106968 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.106635 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="57299a9e-ed21-49ee-acdd-81831c2efd84" containerName="istio-proxy" Nov 02 23:28:57.106968 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.106643 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa4c136-742e-4387-b751-87692a034ba2" containerName="istio-proxy" Nov 02 23:28:57.106968 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.106651 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="afcd99fa-b983-4c6d-83a6-43cd089a8f5e" containerName="istio-proxy" Nov 02 23:28:57.106968 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.106658 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="774684c7-0c28-43c7-bba7-d2c1bcccad16" containerName="istio-proxy" Nov 02 23:28:57.114255 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.114228 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" Nov 02 23:28:57.216318 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.216282 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-adfeab-serving-cert\"" Nov 02 23:28:57.218357 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.218326 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj"] Nov 02 23:28:57.231939 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.231907 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f88718e-12ac-41f3-a041-ad7a676d6e0e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj\" (UID: \"3f88718e-12ac-41f3-a041-ad7a676d6e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" Nov 02 23:28:57.232107 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.231954 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f88718e-12ac-41f3-a041-ad7a676d6e0e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj\" (UID: \"3f88718e-12ac-41f3-a041-ad7a676d6e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" Nov 02 23:28:57.232107 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.232040 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr62k\" (UniqueName: \"kubernetes.io/projected/3f88718e-12ac-41f3-a041-ad7a676d6e0e-kube-api-access-fr62k\") pod \"maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj\" (UID: \"3f88718e-12ac-41f3-a041-ad7a676d6e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" Nov 02 23:28:57.232107 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.232083 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f88718e-12ac-41f3-a041-ad7a676d6e0e-metrics-certs\") pod \"maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj\" (UID: \"3f88718e-12ac-41f3-a041-ad7a676d6e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" Nov 02 23:28:57.332878 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.332838 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f88718e-12ac-41f3-a041-ad7a676d6e0e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj\" (UID: \"3f88718e-12ac-41f3-a041-ad7a676d6e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" Nov 02 23:28:57.333074 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.332894 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f88718e-12ac-41f3-a041-ad7a676d6e0e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj\" (UID: \"3f88718e-12ac-41f3-a041-ad7a676d6e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" Nov 02 23:28:57.333074 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.332922 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr62k\" (UniqueName: \"kubernetes.io/projected/3f88718e-12ac-41f3-a041-ad7a676d6e0e-kube-api-access-fr62k\") pod \"maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj\" (UID: \"3f88718e-12ac-41f3-a041-ad7a676d6e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" Nov 02 23:28:57.333074 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:28:57.333000 5114 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-serving-cert: secret "maas-api-e2e-praxis-b-adfeab-serving-cert" not found Nov 02 23:28:57.333193 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:28:57.333074 5114 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f88718e-12ac-41f3-a041-ad7a676d6e0e-maas-api-tls podName:3f88718e-12ac-41f3-a041-ad7a676d6e0e nodeName:}" failed. No retries permitted until 2026-11-02 23:28:57.833057305 +0000 UTC m=+2413.186090658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3f88718e-12ac-41f3-a041-ad7a676d6e0e-maas-api-tls") pod "maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" (UID: "3f88718e-12ac-41f3-a041-ad7a676d6e0e") : secret "maas-api-e2e-praxis-b-adfeab-serving-cert" not found Nov 02 23:28:57.333193 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.333096 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f88718e-12ac-41f3-a041-ad7a676d6e0e-metrics-certs\") pod \"maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj\" (UID: \"3f88718e-12ac-41f3-a041-ad7a676d6e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" Nov 02 23:28:57.333683 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.333664 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f88718e-12ac-41f3-a041-ad7a676d6e0e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj\" (UID: \"3f88718e-12ac-41f3-a041-ad7a676d6e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" Nov 02 23:28:57.335589 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.335571 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f88718e-12ac-41f3-a041-ad7a676d6e0e-metrics-certs\") pod \"maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj\" (UID: \"3f88718e-12ac-41f3-a041-ad7a676d6e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" Nov 02 23:28:57.394747 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.394673 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr62k\" (UniqueName: \"kubernetes.io/projected/3f88718e-12ac-41f3-a041-ad7a676d6e0e-kube-api-access-fr62k\") pod \"maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj\" (UID: \"3f88718e-12ac-41f3-a041-ad7a676d6e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" Nov 02 23:28:57.840218 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.840175 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f88718e-12ac-41f3-a041-ad7a676d6e0e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj\" (UID: \"3f88718e-12ac-41f3-a041-ad7a676d6e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" Nov 02 23:28:57.842918 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:57.842893 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f88718e-12ac-41f3-a041-ad7a676d6e0e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj\" (UID: \"3f88718e-12ac-41f3-a041-ad7a676d6e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" Nov 02 23:28:58.036434 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:58.036392 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" Nov 02 23:28:58.416235 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:58.416209 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj"] Nov 02 23:28:58.418456 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:28:58.418428 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f88718e_12ac_41f3_a041_ad7a676d6e0e.slice/crio-7c1f6f1a9729ab144ebd48f037eb42e55b84f1c4fe8902b7a839560a424368d6 WatchSource:0}: Error finding container 7c1f6f1a9729ab144ebd48f037eb42e55b84f1c4fe8902b7a839560a424368d6: Status 404 returned error can't find the container with id 7c1f6f1a9729ab144ebd48f037eb42e55b84f1c4fe8902b7a839560a424368d6 Nov 02 23:28:58.420146 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:58.420123 5114 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:28:58.563281 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:58.563245 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" event={"ID":"3f88718e-12ac-41f3-a041-ad7a676d6e0e","Type":"ContainerStarted","Data":"7c1f6f1a9729ab144ebd48f037eb42e55b84f1c4fe8902b7a839560a424368d6"} Nov 02 23:28:59.569699 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:59.569657 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" event={"ID":"3f88718e-12ac-41f3-a041-ad7a676d6e0e","Type":"ContainerStarted","Data":"04ea09333c2c7e00f2e4a2907b74514e2452310f13107aefcbabcfa874299209"} Nov 02 23:28:59.570154 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:59.569847 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" Nov 02 23:28:59.686423 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:28:59.686360 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" podStartSLOduration=1.865008481 podStartE2EDuration="2.686343028s" totalImagesPullingTime="821.334547ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:28:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:28:59.68599173 +0000 UTC m=+2415.039025118" watchObservedRunningTime="2026-11-02 23:28:59.686343028 +0000 UTC m=+2415.039376396" Nov 02 23:29:05.581505 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:29:05.581459 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" Nov 02 23:29:07.585210 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:29:07.585165 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:29:37.584511 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:29:37.584417 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:29:55.021724 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:29:55.021671 5114 patch_prober.go:29] interesting pod/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:29:55.022272 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:29:55.021772 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" podUID="afc01b1c-b566-4d06-bd25-c99d8b60ac3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:29:56.885806 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:29:56.885766 5114 generic.go:356] "Generic (PLEG): container finished" podID="afc01b1c-b566-4d06-bd25-c99d8b60ac3c" containerID="1a19aca8352227c2e610d9e58d332ad83197d6f35c51312905405c7af2c7ab00" exitCode=0 Nov 02 23:29:56.886686 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:29:56.885842 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" event={"ID":"afc01b1c-b566-4d06-bd25-c99d8b60ac3c","Type":"ContainerDied","Data":"1a19aca8352227c2e610d9e58d332ad83197d6f35c51312905405c7af2c7ab00"} Nov 02 23:29:56.886686 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:29:56.886222 5114 scope.go:122] "RemoveContainer" containerID="1a19aca8352227c2e610d9e58d332ad83197d6f35c51312905405c7af2c7ab00" Nov 02 23:29:56.887655 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:29:56.887626 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:29:56.887746 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:29:56.887694 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:29:56.887746 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:29:56.887729 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:29:57.893366 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:29:57.893332 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" event={"ID":"afc01b1c-b566-4d06-bd25-c99d8b60ac3c","Type":"ContainerStarted","Data":"edc8c2aafdae525b4707f98ffb42c81facfe701d759b799e10d97058529742d0"} Nov 02 23:29:58.452825 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:29:58.452756 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:29:58.453147 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:29:58.453122 5114 patch_prober.go:29] interesting pod/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 23:29:58.453217 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:29:58.453175 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" podUID="afc01b1c-b566-4d06-bd25-c99d8b60ac3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 23:29:59.453222 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:29:59.453176 5114 patch_prober.go:29] interesting pod/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 23:29:59.453685 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:29:59.453270 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" podUID="afc01b1c-b566-4d06-bd25-c99d8b60ac3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 23:30:00.170142 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.170087 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b"] Nov 02 23:30:00.170931 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.170903 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="57299a9e-ed21-49ee-acdd-81831c2efd84" containerName="istio-proxy" Nov 02 23:30:00.174756 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.174731 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b" Nov 02 23:30:00.176477 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.176447 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw"] Nov 02 23:30:00.180145 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.180120 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4"] Nov 02 23:30:00.180296 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.180265 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw" Nov 02 23:30:00.183933 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.183911 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4" Nov 02 23:30:00.224604 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.224558 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd"] Nov 02 23:30:00.235675 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.235640 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd" Nov 02 23:30:00.236111 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.236092 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r"] Nov 02 23:30:00.243043 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.243009 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b"] Nov 02 23:30:00.243387 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.243366 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r" Nov 02 23:30:00.265275 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.265232 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbdnf\" (UniqueName: \"kubernetes.io/projected/fa2d7e3f-05c4-4628-9f7e-5e463a12aba4-kube-api-access-cbdnf\") pod \"maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b\" (UID: \"fa2d7e3f-05c4-4628-9f7e-5e463a12aba4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b" Nov 02 23:30:00.265596 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.265577 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c70b5ded-db2a-4965-b6d6-f9121a80ea2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw\" (UID: \"c70b5ded-db2a-4965-b6d6-f9121a80ea2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw" Nov 02 23:30:00.265769 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.265758 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa2d7e3f-05c4-4628-9f7e-5e463a12aba4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b\" (UID: \"fa2d7e3f-05c4-4628-9f7e-5e463a12aba4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b" Nov 02 23:30:00.265880 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.265866 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e148669-1723-42a4-ba95-496f77089093-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4\" (UID: \"8e148669-1723-42a4-ba95-496f77089093\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4" Nov 02 23:30:00.265974 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.265959 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkqk2\" (UniqueName: \"kubernetes.io/projected/8e148669-1723-42a4-ba95-496f77089093-kube-api-access-gkqk2\") pod \"maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4\" (UID: \"8e148669-1723-42a4-ba95-496f77089093\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4" Nov 02 23:30:00.266149 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.266131 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfgvh\" (UniqueName: \"kubernetes.io/projected/c70b5ded-db2a-4965-b6d6-f9121a80ea2c-kube-api-access-sfgvh\") pod \"maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw\" (UID: \"c70b5ded-db2a-4965-b6d6-f9121a80ea2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw" Nov 02 23:30:00.270386 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.270361 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:30:00.274417 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.274387 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw"] Nov 02 23:30:00.283039 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.282989 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4"] Nov 02 23:30:00.283298 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.283279 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd"] Nov 02 23:30:00.287525 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.287495 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r"] Nov 02 23:30:00.367503 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.367445 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/880cd41f-4818-4140-bc4d-6f8676e2feff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r\" (UID: \"880cd41f-4818-4140-bc4d-6f8676e2feff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r" Nov 02 23:30:00.367729 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.367524 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45d8aa7-3576-4203-bc85-9274ec4d6874-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd\" (UID: \"f45d8aa7-3576-4203-bc85-9274ec4d6874\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd" Nov 02 23:30:00.367729 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.367622 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa2d7e3f-05c4-4628-9f7e-5e463a12aba4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b\" (UID: \"fa2d7e3f-05c4-4628-9f7e-5e463a12aba4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b" Nov 02 23:30:00.367729 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.367659 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e148669-1723-42a4-ba95-496f77089093-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4\" (UID: \"8e148669-1723-42a4-ba95-496f77089093\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4" Nov 02 23:30:00.367729 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.367691 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkqk2\" (UniqueName: \"kubernetes.io/projected/8e148669-1723-42a4-ba95-496f77089093-kube-api-access-gkqk2\") pod \"maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4\" (UID: \"8e148669-1723-42a4-ba95-496f77089093\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4" Nov 02 23:30:00.367937 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.367733 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8dvw\" (UniqueName: \"kubernetes.io/projected/880cd41f-4818-4140-bc4d-6f8676e2feff-kube-api-access-w8dvw\") pod \"maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r\" (UID: \"880cd41f-4818-4140-bc4d-6f8676e2feff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r" Nov 02 23:30:00.367937 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.367764 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfgvh\" (UniqueName: \"kubernetes.io/projected/c70b5ded-db2a-4965-b6d6-f9121a80ea2c-kube-api-access-sfgvh\") pod \"maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw\" (UID: \"c70b5ded-db2a-4965-b6d6-f9121a80ea2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw" Nov 02 23:30:00.367937 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.367863 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phmgh\" (UniqueName: \"kubernetes.io/projected/f45d8aa7-3576-4203-bc85-9274ec4d6874-kube-api-access-phmgh\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd\" (UID: \"f45d8aa7-3576-4203-bc85-9274ec4d6874\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd" Nov 02 23:30:00.367937 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.367930 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbdnf\" (UniqueName: \"kubernetes.io/projected/fa2d7e3f-05c4-4628-9f7e-5e463a12aba4-kube-api-access-cbdnf\") pod \"maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b\" (UID: \"fa2d7e3f-05c4-4628-9f7e-5e463a12aba4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b" Nov 02 23:30:00.368129 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.368020 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c70b5ded-db2a-4965-b6d6-f9121a80ea2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw\" (UID: \"c70b5ded-db2a-4965-b6d6-f9121a80ea2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw" Nov 02 23:30:00.368432 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.368412 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e148669-1723-42a4-ba95-496f77089093-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4\" (UID: \"8e148669-1723-42a4-ba95-496f77089093\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4" Nov 02 23:30:00.368510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.368438 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa2d7e3f-05c4-4628-9f7e-5e463a12aba4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b\" (UID: \"fa2d7e3f-05c4-4628-9f7e-5e463a12aba4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b" Nov 02 23:30:00.368677 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.368659 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c70b5ded-db2a-4965-b6d6-f9121a80ea2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw\" (UID: \"c70b5ded-db2a-4965-b6d6-f9121a80ea2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw" Nov 02 23:30:00.434231 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.434112 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbdnf\" (UniqueName: \"kubernetes.io/projected/fa2d7e3f-05c4-4628-9f7e-5e463a12aba4-kube-api-access-cbdnf\") pod \"maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b\" (UID: \"fa2d7e3f-05c4-4628-9f7e-5e463a12aba4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b" Nov 02 23:30:00.434703 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.434673 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfgvh\" (UniqueName: \"kubernetes.io/projected/c70b5ded-db2a-4965-b6d6-f9121a80ea2c-kube-api-access-sfgvh\") pod \"maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw\" (UID: \"c70b5ded-db2a-4965-b6d6-f9121a80ea2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw" Nov 02 23:30:00.435110 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.435086 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkqk2\" (UniqueName: \"kubernetes.io/projected/8e148669-1723-42a4-ba95-496f77089093-kube-api-access-gkqk2\") pod \"maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4\" (UID: \"8e148669-1723-42a4-ba95-496f77089093\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4" Nov 02 23:30:00.453211 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.453180 5114 patch_prober.go:29] interesting pod/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 23:30:00.453379 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.453234 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" podUID="afc01b1c-b566-4d06-bd25-c99d8b60ac3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 23:30:00.469196 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.469154 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8dvw\" (UniqueName: \"kubernetes.io/projected/880cd41f-4818-4140-bc4d-6f8676e2feff-kube-api-access-w8dvw\") pod \"maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r\" (UID: \"880cd41f-4818-4140-bc4d-6f8676e2feff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r" Nov 02 23:30:00.469387 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.469215 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phmgh\" (UniqueName: \"kubernetes.io/projected/f45d8aa7-3576-4203-bc85-9274ec4d6874-kube-api-access-phmgh\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd\" (UID: \"f45d8aa7-3576-4203-bc85-9274ec4d6874\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd" Nov 02 23:30:00.469387 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.469287 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/880cd41f-4818-4140-bc4d-6f8676e2feff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r\" (UID: \"880cd41f-4818-4140-bc4d-6f8676e2feff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r" Nov 02 23:30:00.469387 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.469312 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45d8aa7-3576-4203-bc85-9274ec4d6874-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd\" (UID: \"f45d8aa7-3576-4203-bc85-9274ec4d6874\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd" Nov 02 23:30:00.469918 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.469899 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/880cd41f-4818-4140-bc4d-6f8676e2feff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r\" (UID: \"880cd41f-4818-4140-bc4d-6f8676e2feff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r" Nov 02 23:30:00.469984 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.469965 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45d8aa7-3576-4203-bc85-9274ec4d6874-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd\" (UID: \"f45d8aa7-3576-4203-bc85-9274ec4d6874\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd" Nov 02 23:30:00.491761 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.491721 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b" Nov 02 23:30:00.501355 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.501225 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw" Nov 02 23:30:00.539300 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.539179 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4" Nov 02 23:30:00.540509 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.540366 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phmgh\" (UniqueName: \"kubernetes.io/projected/f45d8aa7-3576-4203-bc85-9274ec4d6874-kube-api-access-phmgh\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd\" (UID: \"f45d8aa7-3576-4203-bc85-9274ec4d6874\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd" Nov 02 23:30:00.541193 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.541076 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8dvw\" (UniqueName: \"kubernetes.io/projected/880cd41f-4818-4140-bc4d-6f8676e2feff-kube-api-access-w8dvw\") pod \"maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r\" (UID: \"880cd41f-4818-4140-bc4d-6f8676e2feff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r" Nov 02 23:30:00.559164 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.559126 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd" Nov 02 23:30:00.582197 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.582159 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r" Nov 02 23:30:00.934329 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.934290 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b"] Nov 02 23:30:00.937588 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:30:00.937560 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2d7e3f_05c4_4628_9f7e_5e463a12aba4.slice/crio-fc3c2bedeb06df4505e66e74c1a3ec8f74ca74d8a0fc61ff1675d85aad956145 WatchSource:0}: Error finding container fc3c2bedeb06df4505e66e74c1a3ec8f74ca74d8a0fc61ff1675d85aad956145: Status 404 returned error can't find the container with id fc3c2bedeb06df4505e66e74c1a3ec8f74ca74d8a0fc61ff1675d85aad956145 Nov 02 23:30:00.964176 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.964140 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw"] Nov 02 23:30:00.965747 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:30:00.965714 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc70b5ded_db2a_4965_b6d6_f9121a80ea2c.slice/crio-894e2762cc1a48c7c5381bc96065ef8ff143a0fc34034740ff09e421bf11825c WatchSource:0}: Error finding container 894e2762cc1a48c7c5381bc96065ef8ff143a0fc34034740ff09e421bf11825c: Status 404 returned error can't find the container with id 894e2762cc1a48c7c5381bc96065ef8ff143a0fc34034740ff09e421bf11825c Nov 02 23:30:00.967359 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:00.967332 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4"] Nov 02 23:30:00.969616 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:30:00.969590 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e148669_1723_42a4_ba95_496f77089093.slice/crio-d1f1e8e827ca2d065c076577db4d8401c1e7bab2bfa3c9be20688b6da9f13b3a WatchSource:0}: Error finding container d1f1e8e827ca2d065c076577db4d8401c1e7bab2bfa3c9be20688b6da9f13b3a: Status 404 returned error can't find the container with id d1f1e8e827ca2d065c076577db4d8401c1e7bab2bfa3c9be20688b6da9f13b3a Nov 02 23:30:01.020532 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:01.018078 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd"] Nov 02 23:30:01.075216 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:01.075157 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r"] Nov 02 23:30:01.453615 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:01.453574 5114 patch_prober.go:29] interesting pod/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 23:30:01.454296 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:01.454247 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" podUID="afc01b1c-b566-4d06-bd25-c99d8b60ac3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 23:30:01.943642 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:01.943433 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b" event={"ID":"fa2d7e3f-05c4-4628-9f7e-5e463a12aba4","Type":"ContainerStarted","Data":"860ed9d5cc1f8f25311375fbeaf260677f801f048bb79784b86cdb9cfd9fe41d"} Nov 02 23:30:01.943877 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:01.943855 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b" event={"ID":"fa2d7e3f-05c4-4628-9f7e-5e463a12aba4","Type":"ContainerStarted","Data":"fc3c2bedeb06df4505e66e74c1a3ec8f74ca74d8a0fc61ff1675d85aad956145"} Nov 02 23:30:01.945006 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:01.944977 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd" event={"ID":"f45d8aa7-3576-4203-bc85-9274ec4d6874","Type":"ContainerStarted","Data":"248e27e0aeeb4682f782fc5ce5a3100c5632738c8ad48669dee4e3508f5f560a"} Nov 02 23:30:01.945354 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:01.945332 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd" event={"ID":"f45d8aa7-3576-4203-bc85-9274ec4d6874","Type":"ContainerStarted","Data":"1b91bc5c342b4362948380c580188cd2075622621c87cf3247886dd3ff3cf7b8"} Nov 02 23:30:01.946528 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:01.946507 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw" event={"ID":"c70b5ded-db2a-4965-b6d6-f9121a80ea2c","Type":"ContainerStarted","Data":"dcb63f723a0fdfdb956999de8b5a5859c8a45d18b11e39cf6c822b8b05c4dd9b"} Nov 02 23:30:01.946597 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:01.946541 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw" event={"ID":"c70b5ded-db2a-4965-b6d6-f9121a80ea2c","Type":"ContainerStarted","Data":"894e2762cc1a48c7c5381bc96065ef8ff143a0fc34034740ff09e421bf11825c"} Nov 02 23:30:01.948010 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:01.947992 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r" event={"ID":"880cd41f-4818-4140-bc4d-6f8676e2feff","Type":"ContainerStarted","Data":"c37bd2115b11d0b0f4817db751e2b7564b3ec2301e4d241516e92a9bb23cec01"} Nov 02 23:30:01.948343 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:01.948325 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r" event={"ID":"880cd41f-4818-4140-bc4d-6f8676e2feff","Type":"ContainerStarted","Data":"9392ba322198e5c7ddfacec919e57dc07a9716b9bea1bc95ea4212586b0b4554"} Nov 02 23:30:01.949461 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:01.949430 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4" event={"ID":"8e148669-1723-42a4-ba95-496f77089093","Type":"ContainerStarted","Data":"b1c750a16d26a6cc759c7e2ce7403c98fad0235059c0f29465234a0db709a86e"} Nov 02 23:30:01.949556 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:01.949496 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4" event={"ID":"8e148669-1723-42a4-ba95-496f77089093","Type":"ContainerStarted","Data":"d1f1e8e827ca2d065c076577db4d8401c1e7bab2bfa3c9be20688b6da9f13b3a"} Nov 02 23:30:02.058635 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:02.058577 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b" podStartSLOduration=2.05856115 podStartE2EDuration="2.05856115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:02.057577304 +0000 UTC m=+2477.410610682" watchObservedRunningTime="2026-11-02 23:30:02.05856115 +0000 UTC m=+2477.411594525" Nov 02 23:30:02.180613 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:02.180554 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd" podStartSLOduration=2.18053698 podStartE2EDuration="2.18053698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:02.178322768 +0000 UTC m=+2477.531356147" watchObservedRunningTime="2026-11-02 23:30:02.18053698 +0000 UTC m=+2477.533570355" Nov 02 23:30:02.295338 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:02.295274 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r" podStartSLOduration=2.2952579269999998 podStartE2EDuration="2.295257927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:02.293198353 +0000 UTC m=+2477.646231728" watchObservedRunningTime="2026-11-02 23:30:02.295257927 +0000 UTC m=+2477.648291302" Nov 02 23:30:02.452721 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:02.452681 5114 patch_prober.go:29] interesting pod/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 23:30:02.452902 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:02.452743 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" podUID="afc01b1c-b566-4d06-bd25-c99d8b60ac3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 23:30:02.545182 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:02.545120 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4" podStartSLOduration=2.54510573 podStartE2EDuration="2.54510573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:02.544876644 +0000 UTC m=+2477.897910020" watchObservedRunningTime="2026-11-02 23:30:02.54510573 +0000 UTC m=+2477.898139105" Nov 02 23:30:02.546934 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:02.546894 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw" podStartSLOduration=2.546882223 podStartE2EDuration="2.546882223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:02.430798547 +0000 UTC m=+2477.783831924" watchObservedRunningTime="2026-11-02 23:30:02.546882223 +0000 UTC m=+2477.899915625" Nov 02 23:30:02.957038 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:02.956934 5114 generic.go:356] "Generic (PLEG): container finished" podID="c70b5ded-db2a-4965-b6d6-f9121a80ea2c" containerID="dcb63f723a0fdfdb956999de8b5a5859c8a45d18b11e39cf6c822b8b05c4dd9b" exitCode=60 Nov 02 23:30:02.957213 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:02.957036 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw" event={"ID":"c70b5ded-db2a-4965-b6d6-f9121a80ea2c","Type":"ContainerDied","Data":"dcb63f723a0fdfdb956999de8b5a5859c8a45d18b11e39cf6c822b8b05c4dd9b"} Nov 02 23:30:02.957389 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:02.957367 5114 scope.go:122] "RemoveContainer" containerID="dcb63f723a0fdfdb956999de8b5a5859c8a45d18b11e39cf6c822b8b05c4dd9b" Nov 02 23:30:02.958878 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:02.958857 5114 generic.go:356] "Generic (PLEG): container finished" podID="880cd41f-4818-4140-bc4d-6f8676e2feff" containerID="c37bd2115b11d0b0f4817db751e2b7564b3ec2301e4d241516e92a9bb23cec01" exitCode=60 Nov 02 23:30:02.958989 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:02.958886 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r" event={"ID":"880cd41f-4818-4140-bc4d-6f8676e2feff","Type":"ContainerDied","Data":"c37bd2115b11d0b0f4817db751e2b7564b3ec2301e4d241516e92a9bb23cec01"} Nov 02 23:30:02.959187 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:02.959170 5114 scope.go:122] "RemoveContainer" containerID="c37bd2115b11d0b0f4817db751e2b7564b3ec2301e4d241516e92a9bb23cec01" Nov 02 23:30:02.960825 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:02.960804 5114 generic.go:356] "Generic (PLEG): container finished" podID="8e148669-1723-42a4-ba95-496f77089093" containerID="b1c750a16d26a6cc759c7e2ce7403c98fad0235059c0f29465234a0db709a86e" exitCode=60 Nov 02 23:30:02.960890 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:02.960875 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4" event={"ID":"8e148669-1723-42a4-ba95-496f77089093","Type":"ContainerDied","Data":"b1c750a16d26a6cc759c7e2ce7403c98fad0235059c0f29465234a0db709a86e"} Nov 02 23:30:02.961149 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:02.961135 5114 scope.go:122] "RemoveContainer" containerID="b1c750a16d26a6cc759c7e2ce7403c98fad0235059c0f29465234a0db709a86e" Nov 02 23:30:02.963453 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:02.963419 5114 generic.go:356] "Generic (PLEG): container finished" podID="fa2d7e3f-05c4-4628-9f7e-5e463a12aba4" containerID="860ed9d5cc1f8f25311375fbeaf260677f801f048bb79784b86cdb9cfd9fe41d" exitCode=60 Nov 02 23:30:02.963620 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:02.963508 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b" event={"ID":"fa2d7e3f-05c4-4628-9f7e-5e463a12aba4","Type":"ContainerDied","Data":"860ed9d5cc1f8f25311375fbeaf260677f801f048bb79784b86cdb9cfd9fe41d"} Nov 02 23:30:02.963724 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:02.963707 5114 scope.go:122] "RemoveContainer" containerID="860ed9d5cc1f8f25311375fbeaf260677f801f048bb79784b86cdb9cfd9fe41d" Nov 02 23:30:02.965126 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:02.965106 5114 generic.go:356] "Generic (PLEG): container finished" podID="f45d8aa7-3576-4203-bc85-9274ec4d6874" containerID="248e27e0aeeb4682f782fc5ce5a3100c5632738c8ad48669dee4e3508f5f560a" exitCode=60 Nov 02 23:30:02.965225 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:02.965171 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd" event={"ID":"f45d8aa7-3576-4203-bc85-9274ec4d6874","Type":"ContainerDied","Data":"248e27e0aeeb4682f782fc5ce5a3100c5632738c8ad48669dee4e3508f5f560a"} Nov 02 23:30:02.965411 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:02.965396 5114 scope.go:122] "RemoveContainer" containerID="248e27e0aeeb4682f782fc5ce5a3100c5632738c8ad48669dee4e3508f5f560a" Nov 02 23:30:03.458064 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:03.458030 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:30:03.458364 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:03.458342 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:30:03.459260 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:03.459241 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:30:03.973066 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:03.973030 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4" event={"ID":"8e148669-1723-42a4-ba95-496f77089093","Type":"ContainerStarted","Data":"2412cad855a8aff7b2c205d4f60d4d4b2fb2c3b11791fd47962fd97e721a32c9"} Nov 02 23:30:03.975011 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:03.974973 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b" event={"ID":"fa2d7e3f-05c4-4628-9f7e-5e463a12aba4","Type":"ContainerStarted","Data":"5a4f53b317be0eab3774f781ce54200b61dc18fda1aef94c0d56e9c3d5c7a4b3"} Nov 02 23:30:03.976675 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:03.976648 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd" event={"ID":"f45d8aa7-3576-4203-bc85-9274ec4d6874","Type":"ContainerStarted","Data":"0c7586d245bef9705bf71b3917a04dd8343d0523fcb49659a39b444dd4af8eac"} Nov 02 23:30:03.978551 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:03.978523 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw" event={"ID":"c70b5ded-db2a-4965-b6d6-f9121a80ea2c","Type":"ContainerStarted","Data":"d0e740f7e1141de1c5358ab28579d63734245c246d72dcea74c0b4018a9bbb4c"} Nov 02 23:30:03.980349 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:03.980322 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r" event={"ID":"880cd41f-4818-4140-bc4d-6f8676e2feff","Type":"ContainerStarted","Data":"6a5104ac68ebe44b772abbbd2c48cd99dc1c7bd9837940ed746cc1be603cc97c"} Nov 02 23:30:04.989438 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:04.989399 5114 generic.go:356] "Generic (PLEG): container finished" podID="c70b5ded-db2a-4965-b6d6-f9121a80ea2c" containerID="d0e740f7e1141de1c5358ab28579d63734245c246d72dcea74c0b4018a9bbb4c" exitCode=60 Nov 02 23:30:04.989930 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:04.989473 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw" event={"ID":"c70b5ded-db2a-4965-b6d6-f9121a80ea2c","Type":"ContainerDied","Data":"d0e740f7e1141de1c5358ab28579d63734245c246d72dcea74c0b4018a9bbb4c"} Nov 02 23:30:04.989930 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:04.989554 5114 scope.go:122] "RemoveContainer" containerID="dcb63f723a0fdfdb956999de8b5a5859c8a45d18b11e39cf6c822b8b05c4dd9b" Nov 02 23:30:04.989930 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:04.989825 5114 scope.go:122] "RemoveContainer" containerID="d0e740f7e1141de1c5358ab28579d63734245c246d72dcea74c0b4018a9bbb4c" Nov 02 23:30:04.990159 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:30:04.990135 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw_odh-ai-gateway-infra(c70b5ded-db2a-4965-b6d6-f9121a80ea2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw" podUID="c70b5ded-db2a-4965-b6d6-f9121a80ea2c" Nov 02 23:30:04.991611 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:04.991586 5114 generic.go:356] "Generic (PLEG): container finished" podID="880cd41f-4818-4140-bc4d-6f8676e2feff" containerID="6a5104ac68ebe44b772abbbd2c48cd99dc1c7bd9837940ed746cc1be603cc97c" exitCode=60 Nov 02 23:30:04.991735 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:04.991709 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r" event={"ID":"880cd41f-4818-4140-bc4d-6f8676e2feff","Type":"ContainerDied","Data":"6a5104ac68ebe44b772abbbd2c48cd99dc1c7bd9837940ed746cc1be603cc97c"} Nov 02 23:30:04.991950 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:04.991936 5114 scope.go:122] "RemoveContainer" containerID="6a5104ac68ebe44b772abbbd2c48cd99dc1c7bd9837940ed746cc1be603cc97c" Nov 02 23:30:04.992235 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:30:04.992216 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r_odh-ai-gateway-infra(880cd41f-4818-4140-bc4d-6f8676e2feff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r" podUID="880cd41f-4818-4140-bc4d-6f8676e2feff" Nov 02 23:30:04.993709 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:04.993680 5114 generic.go:356] "Generic (PLEG): container finished" podID="8e148669-1723-42a4-ba95-496f77089093" containerID="2412cad855a8aff7b2c205d4f60d4d4b2fb2c3b11791fd47962fd97e721a32c9" exitCode=60 Nov 02 23:30:04.993831 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:04.993766 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4" event={"ID":"8e148669-1723-42a4-ba95-496f77089093","Type":"ContainerDied","Data":"2412cad855a8aff7b2c205d4f60d4d4b2fb2c3b11791fd47962fd97e721a32c9"} Nov 02 23:30:04.994026 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:04.994005 5114 scope.go:122] "RemoveContainer" containerID="2412cad855a8aff7b2c205d4f60d4d4b2fb2c3b11791fd47962fd97e721a32c9" Nov 02 23:30:04.994287 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:30:04.994267 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4_odh-ai-gateway-infra(8e148669-1723-42a4-ba95-496f77089093)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4" podUID="8e148669-1723-42a4-ba95-496f77089093" Nov 02 23:30:04.996112 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:04.996065 5114 generic.go:356] "Generic (PLEG): container finished" podID="fa2d7e3f-05c4-4628-9f7e-5e463a12aba4" containerID="5a4f53b317be0eab3774f781ce54200b61dc18fda1aef94c0d56e9c3d5c7a4b3" exitCode=60 Nov 02 23:30:04.996215 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:04.996132 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b" event={"ID":"fa2d7e3f-05c4-4628-9f7e-5e463a12aba4","Type":"ContainerDied","Data":"5a4f53b317be0eab3774f781ce54200b61dc18fda1aef94c0d56e9c3d5c7a4b3"} Nov 02 23:30:04.996348 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:04.996328 5114 scope.go:122] "RemoveContainer" containerID="5a4f53b317be0eab3774f781ce54200b61dc18fda1aef94c0d56e9c3d5c7a4b3" Nov 02 23:30:04.996649 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:30:04.996631 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b_odh-ai-gateway-infra(fa2d7e3f-05c4-4628-9f7e-5e463a12aba4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b" podUID="fa2d7e3f-05c4-4628-9f7e-5e463a12aba4" Nov 02 23:30:04.998255 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:04.998234 5114 generic.go:356] "Generic (PLEG): container finished" podID="f45d8aa7-3576-4203-bc85-9274ec4d6874" containerID="0c7586d245bef9705bf71b3917a04dd8343d0523fcb49659a39b444dd4af8eac" exitCode=60 Nov 02 23:30:04.998342 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:04.998308 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd" event={"ID":"f45d8aa7-3576-4203-bc85-9274ec4d6874","Type":"ContainerDied","Data":"0c7586d245bef9705bf71b3917a04dd8343d0523fcb49659a39b444dd4af8eac"} Nov 02 23:30:04.998577 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:04.998561 5114 scope.go:122] "RemoveContainer" containerID="0c7586d245bef9705bf71b3917a04dd8343d0523fcb49659a39b444dd4af8eac" Nov 02 23:30:04.998888 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:30:04.998862 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd_odh-ai-gateway-infra(f45d8aa7-3576-4203-bc85-9274ec4d6874)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd" podUID="f45d8aa7-3576-4203-bc85-9274ec4d6874" Nov 02 23:30:05.002748 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:05.002730 5114 scope.go:122] "RemoveContainer" containerID="c37bd2115b11d0b0f4817db751e2b7564b3ec2301e4d241516e92a9bb23cec01" Nov 02 23:30:05.015579 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:05.015549 5114 scope.go:122] "RemoveContainer" containerID="b1c750a16d26a6cc759c7e2ce7403c98fad0235059c0f29465234a0db709a86e" Nov 02 23:30:05.028743 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:05.028719 5114 scope.go:122] "RemoveContainer" containerID="860ed9d5cc1f8f25311375fbeaf260677f801f048bb79784b86cdb9cfd9fe41d" Nov 02 23:30:05.042428 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:05.042410 5114 scope.go:122] "RemoveContainer" containerID="248e27e0aeeb4682f782fc5ce5a3100c5632738c8ad48669dee4e3508f5f560a" Nov 02 23:30:06.005659 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:06.005622 5114 scope.go:122] "RemoveContainer" containerID="0c7586d245bef9705bf71b3917a04dd8343d0523fcb49659a39b444dd4af8eac" Nov 02 23:30:06.006154 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:30:06.005991 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd_odh-ai-gateway-infra(f45d8aa7-3576-4203-bc85-9274ec4d6874)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd" podUID="f45d8aa7-3576-4203-bc85-9274ec4d6874" Nov 02 23:30:06.007312 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:06.007286 5114 scope.go:122] "RemoveContainer" containerID="d0e740f7e1141de1c5358ab28579d63734245c246d72dcea74c0b4018a9bbb4c" Nov 02 23:30:06.007638 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:30:06.007606 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw_odh-ai-gateway-infra(c70b5ded-db2a-4965-b6d6-f9121a80ea2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw" podUID="c70b5ded-db2a-4965-b6d6-f9121a80ea2c" Nov 02 23:30:06.008853 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:06.008830 5114 scope.go:122] "RemoveContainer" containerID="6a5104ac68ebe44b772abbbd2c48cd99dc1c7bd9837940ed746cc1be603cc97c" Nov 02 23:30:06.009161 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:30:06.009140 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r_odh-ai-gateway-infra(880cd41f-4818-4140-bc4d-6f8676e2feff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r" podUID="880cd41f-4818-4140-bc4d-6f8676e2feff" Nov 02 23:30:06.010395 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:06.010378 5114 scope.go:122] "RemoveContainer" containerID="2412cad855a8aff7b2c205d4f60d4d4b2fb2c3b11791fd47962fd97e721a32c9" Nov 02 23:30:06.010664 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:30:06.010637 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4_odh-ai-gateway-infra(8e148669-1723-42a4-ba95-496f77089093)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4" podUID="8e148669-1723-42a4-ba95-496f77089093" Nov 02 23:30:06.011791 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:06.011773 5114 scope.go:122] "RemoveContainer" containerID="5a4f53b317be0eab3774f781ce54200b61dc18fda1aef94c0d56e9c3d5c7a4b3" Nov 02 23:30:06.012023 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:30:06.012004 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b_odh-ai-gateway-infra(fa2d7e3f-05c4-4628-9f7e-5e463a12aba4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b" podUID="fa2d7e3f-05c4-4628-9f7e-5e463a12aba4" Nov 02 23:30:07.583900 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:30:07.583860 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:30:16.787464 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:16.787425 5114 scope.go:122] "RemoveContainer" containerID="d0e740f7e1141de1c5358ab28579d63734245c246d72dcea74c0b4018a9bbb4c" Nov 02 23:30:17.787262 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:17.787221 5114 scope.go:122] "RemoveContainer" containerID="5a4f53b317be0eab3774f781ce54200b61dc18fda1aef94c0d56e9c3d5c7a4b3" Nov 02 23:30:17.787433 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:17.787273 5114 scope.go:122] "RemoveContainer" containerID="2412cad855a8aff7b2c205d4f60d4d4b2fb2c3b11791fd47962fd97e721a32c9" Nov 02 23:30:18.075813 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:18.075780 5114 generic.go:356] "Generic (PLEG): container finished" podID="c70b5ded-db2a-4965-b6d6-f9121a80ea2c" containerID="e094fa276a8e3a15573cd7498c70db6f449f3ee77c058fe6ffee95bd5e3922c7" exitCode=60 Nov 02 23:30:18.076253 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:18.075855 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw" event={"ID":"c70b5ded-db2a-4965-b6d6-f9121a80ea2c","Type":"ContainerDied","Data":"e094fa276a8e3a15573cd7498c70db6f449f3ee77c058fe6ffee95bd5e3922c7"} Nov 02 23:30:18.076253 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:18.075913 5114 scope.go:122] "RemoveContainer" containerID="d0e740f7e1141de1c5358ab28579d63734245c246d72dcea74c0b4018a9bbb4c" Nov 02 23:30:18.076342 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:18.076295 5114 scope.go:122] "RemoveContainer" containerID="e094fa276a8e3a15573cd7498c70db6f449f3ee77c058fe6ffee95bd5e3922c7" Nov 02 23:30:18.076636 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:30:18.076614 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw_odh-ai-gateway-infra(c70b5ded-db2a-4965-b6d6-f9121a80ea2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw" podUID="c70b5ded-db2a-4965-b6d6-f9121a80ea2c" Nov 02 23:30:18.787238 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:18.787200 5114 scope.go:122] "RemoveContainer" containerID="0c7586d245bef9705bf71b3917a04dd8343d0523fcb49659a39b444dd4af8eac" Nov 02 23:30:18.787437 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:18.787240 5114 scope.go:122] "RemoveContainer" containerID="6a5104ac68ebe44b772abbbd2c48cd99dc1c7bd9837940ed746cc1be603cc97c" Nov 02 23:30:19.083106 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:19.083070 5114 generic.go:356] "Generic (PLEG): container finished" podID="8e148669-1723-42a4-ba95-496f77089093" containerID="799cbb08bc628773a1f0a5e224a61abf5637cbe9d0284c01f4db6d4b9e275cfb" exitCode=60 Nov 02 23:30:19.083594 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:19.083145 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4" event={"ID":"8e148669-1723-42a4-ba95-496f77089093","Type":"ContainerDied","Data":"799cbb08bc628773a1f0a5e224a61abf5637cbe9d0284c01f4db6d4b9e275cfb"} Nov 02 23:30:19.083594 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:19.083213 5114 scope.go:122] "RemoveContainer" containerID="2412cad855a8aff7b2c205d4f60d4d4b2fb2c3b11791fd47962fd97e721a32c9" Nov 02 23:30:19.083695 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:19.083626 5114 scope.go:122] "RemoveContainer" containerID="799cbb08bc628773a1f0a5e224a61abf5637cbe9d0284c01f4db6d4b9e275cfb" Nov 02 23:30:19.083922 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:30:19.083897 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4_odh-ai-gateway-infra(8e148669-1723-42a4-ba95-496f77089093)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4" podUID="8e148669-1723-42a4-ba95-496f77089093" Nov 02 23:30:19.085240 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:19.085217 5114 generic.go:356] "Generic (PLEG): container finished" podID="fa2d7e3f-05c4-4628-9f7e-5e463a12aba4" containerID="eb745bef1b600cdd3f034fbec66185e8f932194653a05c89b63e8880243acd74" exitCode=60 Nov 02 23:30:19.085337 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:19.085276 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b" event={"ID":"fa2d7e3f-05c4-4628-9f7e-5e463a12aba4","Type":"ContainerDied","Data":"eb745bef1b600cdd3f034fbec66185e8f932194653a05c89b63e8880243acd74"} Nov 02 23:30:19.085667 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:19.085653 5114 scope.go:122] "RemoveContainer" containerID="eb745bef1b600cdd3f034fbec66185e8f932194653a05c89b63e8880243acd74" Nov 02 23:30:19.085905 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:30:19.085879 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b_odh-ai-gateway-infra(fa2d7e3f-05c4-4628-9f7e-5e463a12aba4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b" podUID="fa2d7e3f-05c4-4628-9f7e-5e463a12aba4" Nov 02 23:30:19.095195 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:19.095175 5114 scope.go:122] "RemoveContainer" containerID="5a4f53b317be0eab3774f781ce54200b61dc18fda1aef94c0d56e9c3d5c7a4b3" Nov 02 23:30:19.205056 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:19.204966 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw"] Nov 02 23:30:19.367639 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:19.367605 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw" Nov 02 23:30:19.485727 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:19.485632 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c70b5ded-db2a-4965-b6d6-f9121a80ea2c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c70b5ded-db2a-4965-b6d6-f9121a80ea2c-maas-api-service-ca\") pod \"c70b5ded-db2a-4965-b6d6-f9121a80ea2c\" (UID: \"c70b5ded-db2a-4965-b6d6-f9121a80ea2c\") " Nov 02 23:30:19.485913 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:19.485760 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c70b5ded-db2a-4965-b6d6-f9121a80ea2c-kube-api-access-sfgvh\" (UniqueName: \"kubernetes.io/projected/c70b5ded-db2a-4965-b6d6-f9121a80ea2c-kube-api-access-sfgvh\") pod \"c70b5ded-db2a-4965-b6d6-f9121a80ea2c\" (UID: \"c70b5ded-db2a-4965-b6d6-f9121a80ea2c\") " Nov 02 23:30:19.486060 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:19.486035 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c70b5ded-db2a-4965-b6d6-f9121a80ea2c-maas-api-service-ca" pod "c70b5ded-db2a-4965-b6d6-f9121a80ea2c" (UID: "c70b5ded-db2a-4965-b6d6-f9121a80ea2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:19.488734 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:19.488708 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70b5ded-db2a-4965-b6d6-f9121a80ea2c-kube-api-access-sfgvh" pod "c70b5ded-db2a-4965-b6d6-f9121a80ea2c" (UID: "c70b5ded-db2a-4965-b6d6-f9121a80ea2c"). InnerVolumeSpecName "kube-api-access-sfgvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:19.586700 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:19.586659 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfgvh\" (UniqueName: \"kubernetes.io/projected/c70b5ded-db2a-4965-b6d6-f9121a80ea2c-kube-api-access-sfgvh\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:19.586700 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:19.586689 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c70b5ded-db2a-4965-b6d6-f9121a80ea2c-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:20.094515 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.094403 5114 generic.go:356] "Generic (PLEG): container finished" podID="f45d8aa7-3576-4203-bc85-9274ec4d6874" containerID="2753534ae88a170e21c28b00e7e1a3aa5648087fafe1db64ad7bd034b00cf9fc" exitCode=60 Nov 02 23:30:20.094515 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.094475 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd" event={"ID":"f45d8aa7-3576-4203-bc85-9274ec4d6874","Type":"ContainerDied","Data":"2753534ae88a170e21c28b00e7e1a3aa5648087fafe1db64ad7bd034b00cf9fc"} Nov 02 23:30:20.095061 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.094562 5114 scope.go:122] "RemoveContainer" containerID="0c7586d245bef9705bf71b3917a04dd8343d0523fcb49659a39b444dd4af8eac" Nov 02 23:30:20.095061 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.094962 5114 scope.go:122] "RemoveContainer" containerID="2753534ae88a170e21c28b00e7e1a3aa5648087fafe1db64ad7bd034b00cf9fc" Nov 02 23:30:20.095282 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:30:20.095247 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd_odh-ai-gateway-infra(f45d8aa7-3576-4203-bc85-9274ec4d6874)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd" podUID="f45d8aa7-3576-4203-bc85-9274ec4d6874" Nov 02 23:30:20.096260 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.096235 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw" event={"ID":"c70b5ded-db2a-4965-b6d6-f9121a80ea2c","Type":"ContainerDied","Data":"894e2762cc1a48c7c5381bc96065ef8ff143a0fc34034740ff09e421bf11825c"} Nov 02 23:30:20.096334 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.096258 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw" Nov 02 23:30:20.098813 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.098782 5114 generic.go:356] "Generic (PLEG): container finished" podID="880cd41f-4818-4140-bc4d-6f8676e2feff" containerID="c9fc7bf588c11843766c74adf441f8e569d8519834cdf614fb0ef29b89f2d580" exitCode=60 Nov 02 23:30:20.098916 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.098849 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r" event={"ID":"880cd41f-4818-4140-bc4d-6f8676e2feff","Type":"ContainerDied","Data":"c9fc7bf588c11843766c74adf441f8e569d8519834cdf614fb0ef29b89f2d580"} Nov 02 23:30:20.099298 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.099274 5114 scope.go:122] "RemoveContainer" containerID="c9fc7bf588c11843766c74adf441f8e569d8519834cdf614fb0ef29b89f2d580" Nov 02 23:30:20.099610 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:30:20.099588 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r_odh-ai-gateway-infra(880cd41f-4818-4140-bc4d-6f8676e2feff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r" podUID="880cd41f-4818-4140-bc4d-6f8676e2feff" Nov 02 23:30:20.108635 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.108613 5114 scope.go:122] "RemoveContainer" containerID="e094fa276a8e3a15573cd7498c70db6f449f3ee77c058fe6ffee95bd5e3922c7" Nov 02 23:30:20.119699 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.119669 5114 scope.go:122] "RemoveContainer" containerID="6a5104ac68ebe44b772abbbd2c48cd99dc1c7bd9837940ed746cc1be603cc97c" Nov 02 23:30:20.208056 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.208021 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4"] Nov 02 23:30:20.329986 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.329953 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b"] Nov 02 23:30:20.374163 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.374131 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4" Nov 02 23:30:20.434526 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.434473 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw"] Nov 02 23:30:20.438201 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.438172 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894370-99jfw"] Nov 02 23:30:20.496639 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.496604 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b" Nov 02 23:30:20.497079 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.497056 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e148669-1723-42a4-ba95-496f77089093-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e148669-1723-42a4-ba95-496f77089093-maas-api-service-ca\") pod \"8e148669-1723-42a4-ba95-496f77089093\" (UID: \"8e148669-1723-42a4-ba95-496f77089093\") " Nov 02 23:30:20.497222 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.497205 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e148669-1723-42a4-ba95-496f77089093-kube-api-access-gkqk2\" (UniqueName: \"kubernetes.io/projected/8e148669-1723-42a4-ba95-496f77089093-kube-api-access-gkqk2\") pod \"8e148669-1723-42a4-ba95-496f77089093\" (UID: \"8e148669-1723-42a4-ba95-496f77089093\") " Nov 02 23:30:20.501536 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.497558 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e148669-1723-42a4-ba95-496f77089093-maas-api-service-ca" pod "8e148669-1723-42a4-ba95-496f77089093" (UID: "8e148669-1723-42a4-ba95-496f77089093"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:20.501536 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.497765 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e148669-1723-42a4-ba95-496f77089093-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:20.501536 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.500121 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e148669-1723-42a4-ba95-496f77089093-kube-api-access-gkqk2" pod "8e148669-1723-42a4-ba95-496f77089093" (UID: "8e148669-1723-42a4-ba95-496f77089093"). InnerVolumeSpecName "kube-api-access-gkqk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:20.598412 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.598364 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2d7e3f-05c4-4628-9f7e-5e463a12aba4-kube-api-access-cbdnf\" (UniqueName: \"kubernetes.io/projected/fa2d7e3f-05c4-4628-9f7e-5e463a12aba4-kube-api-access-cbdnf\") pod \"fa2d7e3f-05c4-4628-9f7e-5e463a12aba4\" (UID: \"fa2d7e3f-05c4-4628-9f7e-5e463a12aba4\") " Nov 02 23:30:20.598619 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.598557 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa2d7e3f-05c4-4628-9f7e-5e463a12aba4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa2d7e3f-05c4-4628-9f7e-5e463a12aba4-maas-api-service-ca\") pod \"fa2d7e3f-05c4-4628-9f7e-5e463a12aba4\" (UID: \"fa2d7e3f-05c4-4628-9f7e-5e463a12aba4\") " Nov 02 23:30:20.599006 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.598977 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkqk2\" (UniqueName: \"kubernetes.io/projected/8e148669-1723-42a4-ba95-496f77089093-kube-api-access-gkqk2\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:20.599241 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.599219 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2d7e3f-05c4-4628-9f7e-5e463a12aba4-maas-api-service-ca" pod "fa2d7e3f-05c4-4628-9f7e-5e463a12aba4" (UID: "fa2d7e3f-05c4-4628-9f7e-5e463a12aba4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:20.601217 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.601154 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2d7e3f-05c4-4628-9f7e-5e463a12aba4-kube-api-access-cbdnf" pod "fa2d7e3f-05c4-4628-9f7e-5e463a12aba4" (UID: "fa2d7e3f-05c4-4628-9f7e-5e463a12aba4"). InnerVolumeSpecName "kube-api-access-cbdnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:20.700264 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.700228 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbdnf\" (UniqueName: \"kubernetes.io/projected/fa2d7e3f-05c4-4628-9f7e-5e463a12aba4-kube-api-access-cbdnf\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:20.700264 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.700261 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa2d7e3f-05c4-4628-9f7e-5e463a12aba4-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:20.794343 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:20.794311 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c70b5ded-db2a-4965-b6d6-f9121a80ea2c" path="/var/lib/kubelet/pods/c70b5ded-db2a-4965-b6d6-f9121a80ea2c/volumes" Nov 02 23:30:21.109383 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.109341 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4" event={"ID":"8e148669-1723-42a4-ba95-496f77089093","Type":"ContainerDied","Data":"d1f1e8e827ca2d065c076577db4d8401c1e7bab2bfa3c9be20688b6da9f13b3a"} Nov 02 23:30:21.109383 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.109369 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4" Nov 02 23:30:21.109956 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.109419 5114 scope.go:122] "RemoveContainer" containerID="799cbb08bc628773a1f0a5e224a61abf5637cbe9d0284c01f4db6d4b9e275cfb" Nov 02 23:30:21.111421 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.111356 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b" event={"ID":"fa2d7e3f-05c4-4628-9f7e-5e463a12aba4","Type":"ContainerDied","Data":"fc3c2bedeb06df4505e66e74c1a3ec8f74ca74d8a0fc61ff1675d85aad956145"} Nov 02 23:30:21.111681 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.111659 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b" Nov 02 23:30:21.126360 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.126334 5114 scope.go:122] "RemoveContainer" containerID="eb745bef1b600cdd3f034fbec66185e8f932194653a05c89b63e8880243acd74" Nov 02 23:30:21.220035 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.220001 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd"] Nov 02 23:30:21.272039 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.272006 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4"] Nov 02 23:30:21.276367 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.276338 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-014ab4-29894370-kxxf4"] Nov 02 23:30:21.333594 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.333560 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r"] Nov 02 23:30:21.381602 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.381517 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd" Nov 02 23:30:21.384675 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.384647 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b"] Nov 02 23:30:21.390403 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.390370 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1cb1f8-29894370-xlg9b"] Nov 02 23:30:21.491926 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.491897 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r" Nov 02 23:30:21.506595 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.506562 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f45d8aa7-3576-4203-bc85-9274ec4d6874-kube-api-access-phmgh\" (UniqueName: \"kubernetes.io/projected/f45d8aa7-3576-4203-bc85-9274ec4d6874-kube-api-access-phmgh\") pod \"f45d8aa7-3576-4203-bc85-9274ec4d6874\" (UID: \"f45d8aa7-3576-4203-bc85-9274ec4d6874\") " Nov 02 23:30:21.506595 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.506609 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f45d8aa7-3576-4203-bc85-9274ec4d6874-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45d8aa7-3576-4203-bc85-9274ec4d6874-maas-api-service-ca\") pod \"f45d8aa7-3576-4203-bc85-9274ec4d6874\" (UID: \"f45d8aa7-3576-4203-bc85-9274ec4d6874\") " Nov 02 23:30:21.507003 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.506974 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f45d8aa7-3576-4203-bc85-9274ec4d6874-maas-api-service-ca" pod "f45d8aa7-3576-4203-bc85-9274ec4d6874" (UID: "f45d8aa7-3576-4203-bc85-9274ec4d6874"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:21.509301 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.509272 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45d8aa7-3576-4203-bc85-9274ec4d6874-kube-api-access-phmgh" pod "f45d8aa7-3576-4203-bc85-9274ec4d6874" (UID: "f45d8aa7-3576-4203-bc85-9274ec4d6874"). InnerVolumeSpecName "kube-api-access-phmgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:21.607731 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.607697 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/880cd41f-4818-4140-bc4d-6f8676e2feff-kube-api-access-w8dvw\" (UniqueName: \"kubernetes.io/projected/880cd41f-4818-4140-bc4d-6f8676e2feff-kube-api-access-w8dvw\") pod \"880cd41f-4818-4140-bc4d-6f8676e2feff\" (UID: \"880cd41f-4818-4140-bc4d-6f8676e2feff\") " Nov 02 23:30:21.607958 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.607938 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/880cd41f-4818-4140-bc4d-6f8676e2feff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/880cd41f-4818-4140-bc4d-6f8676e2feff-maas-api-service-ca\") pod \"880cd41f-4818-4140-bc4d-6f8676e2feff\" (UID: \"880cd41f-4818-4140-bc4d-6f8676e2feff\") " Nov 02 23:30:21.608263 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.608240 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/880cd41f-4818-4140-bc4d-6f8676e2feff-maas-api-service-ca" pod "880cd41f-4818-4140-bc4d-6f8676e2feff" (UID: "880cd41f-4818-4140-bc4d-6f8676e2feff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:21.608400 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.608385 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phmgh\" (UniqueName: \"kubernetes.io/projected/f45d8aa7-3576-4203-bc85-9274ec4d6874-kube-api-access-phmgh\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:21.608463 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.608407 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/880cd41f-4818-4140-bc4d-6f8676e2feff-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:21.608463 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.608424 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45d8aa7-3576-4203-bc85-9274ec4d6874-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:21.610150 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.610128 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880cd41f-4818-4140-bc4d-6f8676e2feff-kube-api-access-w8dvw" pod "880cd41f-4818-4140-bc4d-6f8676e2feff" (UID: "880cd41f-4818-4140-bc4d-6f8676e2feff"). InnerVolumeSpecName "kube-api-access-w8dvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:21.710006 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:21.709914 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8dvw\" (UniqueName: \"kubernetes.io/projected/880cd41f-4818-4140-bc4d-6f8676e2feff-kube-api-access-w8dvw\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:22.157081 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:22.157047 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd" Nov 02 23:30:22.157619 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:22.157047 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd" event={"ID":"f45d8aa7-3576-4203-bc85-9274ec4d6874","Type":"ContainerDied","Data":"1b91bc5c342b4362948380c580188cd2075622621c87cf3247886dd3ff3cf7b8"} Nov 02 23:30:22.157619 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:22.157231 5114 scope.go:122] "RemoveContainer" containerID="2753534ae88a170e21c28b00e7e1a3aa5648087fafe1db64ad7bd034b00cf9fc" Nov 02 23:30:22.158940 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:22.158672 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r" Nov 02 23:30:22.159037 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:22.158672 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r" event={"ID":"880cd41f-4818-4140-bc4d-6f8676e2feff","Type":"ContainerDied","Data":"9392ba322198e5c7ddfacec919e57dc07a9716b9bea1bc95ea4212586b0b4554"} Nov 02 23:30:22.180557 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:22.178885 5114 scope.go:122] "RemoveContainer" containerID="c9fc7bf588c11843766c74adf441f8e569d8519834cdf614fb0ef29b89f2d580" Nov 02 23:30:22.321190 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:22.321140 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd"] Nov 02 23:30:22.324094 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:22.324068 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894370-fwdfd"] Nov 02 23:30:22.432387 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:22.432302 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r"] Nov 02 23:30:22.437309 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:22.437279 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-818fe8-29894370-shr8r"] Nov 02 23:30:22.794225 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:22.794194 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="880cd41f-4818-4140-bc4d-6f8676e2feff" path="/var/lib/kubelet/pods/880cd41f-4818-4140-bc4d-6f8676e2feff/volumes" Nov 02 23:30:22.794627 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:22.794613 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e148669-1723-42a4-ba95-496f77089093" path="/var/lib/kubelet/pods/8e148669-1723-42a4-ba95-496f77089093/volumes" Nov 02 23:30:22.794962 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:22.794949 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f45d8aa7-3576-4203-bc85-9274ec4d6874" path="/var/lib/kubelet/pods/f45d8aa7-3576-4203-bc85-9274ec4d6874/volumes" Nov 02 23:30:22.795628 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:22.795614 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa2d7e3f-05c4-4628-9f7e-5e463a12aba4" path="/var/lib/kubelet/pods/fa2d7e3f-05c4-4628-9f7e-5e463a12aba4/volumes" Nov 02 23:30:26.438404 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:26.438366 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx"] Nov 02 23:30:26.438864 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:26.438659 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" podUID="49b8de5c-62d9-4246-a307-796410ed7bc0" containerName="istio-proxy" containerID="cri-o://1ff52d90e3618a92c6fe8c9a85d49706b5d89b57f3623f9c52ccb6b4ece91c1d" gracePeriod=30 Nov 02 23:30:28.122336 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:28.122298 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h"] Nov 02 23:30:28.122839 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:28.122580 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" podUID="336646a8-fbe7-4780-85b8-116a012ea309" containerName="istio-proxy" containerID="cri-o://345384af6035acab85499fd1250cf64f75bc32512ddc8bd04197e4d57ec9b8ac" gracePeriod=30 Nov 02 23:30:28.599571 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:28.599529 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:30:28.599758 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:28.599607 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" podUID="49b8de5c-62d9-4246-a307-796410ed7bc0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:30:29.226272 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.226233 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6476e8ca-fk898"] Nov 02 23:30:29.227191 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227169 5114 cpu_manager.go:414] "containerMap: removing container" podUID="880cd41f-4818-4140-bc4d-6f8676e2feff" containerName="cleanup" Nov 02 23:30:29.227245 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227196 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="880cd41f-4818-4140-bc4d-6f8676e2feff" containerName="cleanup" Nov 02 23:30:29.227245 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227211 5114 cpu_manager.go:414] "containerMap: removing container" podUID="8e148669-1723-42a4-ba95-496f77089093" containerName="cleanup" Nov 02 23:30:29.227245 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227219 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e148669-1723-42a4-ba95-496f77089093" containerName="cleanup" Nov 02 23:30:29.227336 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227238 5114 cpu_manager.go:414] "containerMap: removing container" podUID="c70b5ded-db2a-4965-b6d6-f9121a80ea2c" containerName="cleanup" Nov 02 23:30:29.227336 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227276 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c70b5ded-db2a-4965-b6d6-f9121a80ea2c" containerName="cleanup" Nov 02 23:30:29.227336 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227300 5114 cpu_manager.go:414] "containerMap: removing container" podUID="c70b5ded-db2a-4965-b6d6-f9121a80ea2c" containerName="cleanup" Nov 02 23:30:29.227336 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227328 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c70b5ded-db2a-4965-b6d6-f9121a80ea2c" containerName="cleanup" Nov 02 23:30:29.227472 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227339 5114 cpu_manager.go:414] "containerMap: removing container" podUID="c70b5ded-db2a-4965-b6d6-f9121a80ea2c" containerName="cleanup" Nov 02 23:30:29.227472 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227346 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c70b5ded-db2a-4965-b6d6-f9121a80ea2c" containerName="cleanup" Nov 02 23:30:29.227472 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227399 5114 cpu_manager.go:414] "containerMap: removing container" podUID="fa2d7e3f-05c4-4628-9f7e-5e463a12aba4" containerName="cleanup" Nov 02 23:30:29.227472 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227406 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2d7e3f-05c4-4628-9f7e-5e463a12aba4" containerName="cleanup" Nov 02 23:30:29.227472 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227418 5114 cpu_manager.go:414] "containerMap: removing container" podUID="fa2d7e3f-05c4-4628-9f7e-5e463a12aba4" containerName="cleanup" Nov 02 23:30:29.227472 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227425 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2d7e3f-05c4-4628-9f7e-5e463a12aba4" containerName="cleanup" Nov 02 23:30:29.227472 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227445 5114 cpu_manager.go:414] "containerMap: removing container" podUID="f45d8aa7-3576-4203-bc85-9274ec4d6874" containerName="cleanup" Nov 02 23:30:29.227472 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227452 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45d8aa7-3576-4203-bc85-9274ec4d6874" containerName="cleanup" Nov 02 23:30:29.227472 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227472 5114 cpu_manager.go:414] "containerMap: removing container" podUID="8e148669-1723-42a4-ba95-496f77089093" containerName="cleanup" Nov 02 23:30:29.227472 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227496 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e148669-1723-42a4-ba95-496f77089093" containerName="cleanup" Nov 02 23:30:29.227799 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227506 5114 cpu_manager.go:414] "containerMap: removing container" podUID="880cd41f-4818-4140-bc4d-6f8676e2feff" containerName="cleanup" Nov 02 23:30:29.227799 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227513 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="880cd41f-4818-4140-bc4d-6f8676e2feff" containerName="cleanup" Nov 02 23:30:29.227799 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227529 5114 cpu_manager.go:414] "containerMap: removing container" podUID="880cd41f-4818-4140-bc4d-6f8676e2feff" containerName="cleanup" Nov 02 23:30:29.227799 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227535 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="880cd41f-4818-4140-bc4d-6f8676e2feff" containerName="cleanup" Nov 02 23:30:29.227799 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227561 5114 cpu_manager.go:414] "containerMap: removing container" podUID="8e148669-1723-42a4-ba95-496f77089093" containerName="cleanup" Nov 02 23:30:29.227799 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227568 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e148669-1723-42a4-ba95-496f77089093" containerName="cleanup" Nov 02 23:30:29.227799 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227590 5114 cpu_manager.go:414] "containerMap: removing container" podUID="f45d8aa7-3576-4203-bc85-9274ec4d6874" containerName="cleanup" Nov 02 23:30:29.227799 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227598 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45d8aa7-3576-4203-bc85-9274ec4d6874" containerName="cleanup" Nov 02 23:30:29.227799 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227791 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2d7e3f-05c4-4628-9f7e-5e463a12aba4" containerName="cleanup" Nov 02 23:30:29.228055 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227803 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2d7e3f-05c4-4628-9f7e-5e463a12aba4" containerName="cleanup" Nov 02 23:30:29.228055 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227818 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45d8aa7-3576-4203-bc85-9274ec4d6874" containerName="cleanup" Nov 02 23:30:29.228055 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227826 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e148669-1723-42a4-ba95-496f77089093" containerName="cleanup" Nov 02 23:30:29.228055 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227833 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="880cd41f-4818-4140-bc4d-6f8676e2feff" containerName="cleanup" Nov 02 23:30:29.228055 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227843 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="880cd41f-4818-4140-bc4d-6f8676e2feff" containerName="cleanup" Nov 02 23:30:29.228055 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227859 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45d8aa7-3576-4203-bc85-9274ec4d6874" containerName="cleanup" Nov 02 23:30:29.228055 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227872 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e148669-1723-42a4-ba95-496f77089093" containerName="cleanup" Nov 02 23:30:29.228055 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227882 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e148669-1723-42a4-ba95-496f77089093" containerName="cleanup" Nov 02 23:30:29.228055 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227898 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="c70b5ded-db2a-4965-b6d6-f9121a80ea2c" containerName="cleanup" Nov 02 23:30:29.228055 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227907 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="c70b5ded-db2a-4965-b6d6-f9121a80ea2c" containerName="cleanup" Nov 02 23:30:29.228055 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.227920 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="c70b5ded-db2a-4965-b6d6-f9121a80ea2c" containerName="cleanup" Nov 02 23:30:29.233382 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.233340 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e8ca-fk898" Nov 02 23:30:29.284138 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.284104 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7"] Nov 02 23:30:29.288575 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.288538 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6476e8ca-fk898"] Nov 02 23:30:29.289090 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.289062 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sz5q\" (UniqueName: \"kubernetes.io/projected/28b616d3-66da-4466-8880-de11323f6ad9-kube-api-access-6sz5q\") pod \"maas-api-revoke-sub-6476e8ca-fk898\" (UID: \"28b616d3-66da-4466-8880-de11323f6ad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e8ca-fk898" Nov 02 23:30:29.289371 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.289347 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b616d3-66da-4466-8880-de11323f6ad9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6476e8ca-fk898\" (UID: \"28b616d3-66da-4466-8880-de11323f6ad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e8ca-fk898" Nov 02 23:30:29.289498 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.289381 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b616d3-66da-4466-8880-de11323f6ad9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6476e8ca-fk898\" (UID: \"28b616d3-66da-4466-8880-de11323f6ad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e8ca-fk898" Nov 02 23:30:29.331973 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.331937 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:30:29.331973 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.331946 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:30:29.390753 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.390717 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b616d3-66da-4466-8880-de11323f6ad9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6476e8ca-fk898\" (UID: \"28b616d3-66da-4466-8880-de11323f6ad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e8ca-fk898" Nov 02 23:30:29.390753 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.390751 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b616d3-66da-4466-8880-de11323f6ad9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6476e8ca-fk898\" (UID: \"28b616d3-66da-4466-8880-de11323f6ad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e8ca-fk898" Nov 02 23:30:29.390976 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.390769 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sz5q\" (UniqueName: \"kubernetes.io/projected/28b616d3-66da-4466-8880-de11323f6ad9-kube-api-access-6sz5q\") pod \"maas-api-revoke-sub-6476e8ca-fk898\" (UID: \"28b616d3-66da-4466-8880-de11323f6ad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e8ca-fk898" Nov 02 23:30:29.391439 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.391411 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b616d3-66da-4466-8880-de11323f6ad9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6476e8ca-fk898\" (UID: \"28b616d3-66da-4466-8880-de11323f6ad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e8ca-fk898" Nov 02 23:30:29.394012 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.393982 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b616d3-66da-4466-8880-de11323f6ad9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6476e8ca-fk898\" (UID: \"28b616d3-66da-4466-8880-de11323f6ad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e8ca-fk898" Nov 02 23:30:29.451872 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.451829 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sz5q\" (UniqueName: \"kubernetes.io/projected/28b616d3-66da-4466-8880-de11323f6ad9-kube-api-access-6sz5q\") pod \"maas-api-revoke-sub-6476e8ca-fk898\" (UID: \"28b616d3-66da-4466-8880-de11323f6ad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e8ca-fk898" Nov 02 23:30:29.546535 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.546464 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e8ca-fk898" Nov 02 23:30:29.923757 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:29.923725 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6476e8ca-fk898"] Nov 02 23:30:29.926677 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:30:29.926645 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b616d3_66da_4466_8880_de11323f6ad9.slice/crio-2bd66722e94388123f77812f666dfb71c74f6958f22e8142264357e775e4738f WatchSource:0}: Error finding container 2bd66722e94388123f77812f666dfb71c74f6958f22e8142264357e775e4738f: Status 404 returned error can't find the container with id 2bd66722e94388123f77812f666dfb71c74f6958f22e8142264357e775e4738f Nov 02 23:30:30.242538 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:30.242501 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e8ca-fk898" event={"ID":"28b616d3-66da-4466-8880-de11323f6ad9","Type":"ContainerStarted","Data":"4a1f1f4bbc4b85802ebd199e480a7f37df08bd3df9470182c1928f52132a738b"} Nov 02 23:30:30.243019 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:30.242563 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e8ca-fk898" event={"ID":"28b616d3-66da-4466-8880-de11323f6ad9","Type":"ContainerStarted","Data":"2bd66722e94388123f77812f666dfb71c74f6958f22e8142264357e775e4738f"} Nov 02 23:30:30.357685 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:30.357624 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e8ca-fk898" podStartSLOduration=1.357609756 podStartE2EDuration="1.357609756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:30.356805473 +0000 UTC m=+2505.709838868" watchObservedRunningTime="2026-11-02 23:30:30.357609756 +0000 UTC m=+2505.710643112" Nov 02 23:30:31.249340 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.249235 5114 generic.go:356] "Generic (PLEG): container finished" podID="28b616d3-66da-4466-8880-de11323f6ad9" containerID="4a1f1f4bbc4b85802ebd199e480a7f37df08bd3df9470182c1928f52132a738b" exitCode=0 Nov 02 23:30:31.249340 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.249283 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e8ca-fk898" event={"ID":"28b616d3-66da-4466-8880-de11323f6ad9","Type":"ContainerDied","Data":"4a1f1f4bbc4b85802ebd199e480a7f37df08bd3df9470182c1928f52132a738b"} Nov 02 23:30:31.704502 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.704461 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:30:31.821932 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.821832 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-token\" (UniqueName: \"kubernetes.io/projected/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-token\") pod \"49b8de5c-62d9-4246-a307-796410ed7bc0\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " Nov 02 23:30:31.821932 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.821882 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-podinfo\") pod \"49b8de5c-62d9-4246-a307-796410ed7bc0\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " Nov 02 23:30:31.821932 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.821932 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-workload-socket\") pod \"49b8de5c-62d9-4246-a307-796410ed7bc0\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " Nov 02 23:30:31.822228 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.821965 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49b8de5c-62d9-4246-a307-796410ed7bc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49b8de5c-62d9-4246-a307-796410ed7bc0-istiod-ca-cert\") pod \"49b8de5c-62d9-4246-a307-796410ed7bc0\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " Nov 02 23:30:31.822228 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.822000 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-envoy\") pod \"49b8de5c-62d9-4246-a307-796410ed7bc0\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " Nov 02 23:30:31.822228 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.822130 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-data\") pod \"49b8de5c-62d9-4246-a307-796410ed7bc0\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " Nov 02 23:30:31.822228 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.822171 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-credential-socket\") pod \"49b8de5c-62d9-4246-a307-796410ed7bc0\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " Nov 02 23:30:31.822438 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.822246 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49b8de5c-62d9-4246-a307-796410ed7bc0-kube-api-access-lwp6h\" (UniqueName: \"kubernetes.io/projected/49b8de5c-62d9-4246-a307-796410ed7bc0-kube-api-access-lwp6h\") pod \"49b8de5c-62d9-4246-a307-796410ed7bc0\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " Nov 02 23:30:31.822438 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.822304 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-workload-certs\") pod \"49b8de5c-62d9-4246-a307-796410ed7bc0\" (UID: \"49b8de5c-62d9-4246-a307-796410ed7bc0\") " Nov 02 23:30:31.822438 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.822303 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-workload-socket" pod "49b8de5c-62d9-4246-a307-796410ed7bc0" (UID: "49b8de5c-62d9-4246-a307-796410ed7bc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:30:31.822438 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.822410 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49b8de5c-62d9-4246-a307-796410ed7bc0-istiod-ca-cert" pod "49b8de5c-62d9-4246-a307-796410ed7bc0" (UID: "49b8de5c-62d9-4246-a307-796410ed7bc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:31.822691 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.822655 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-workload-certs" pod "49b8de5c-62d9-4246-a307-796410ed7bc0" (UID: "49b8de5c-62d9-4246-a307-796410ed7bc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:30:31.822785 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.822754 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-credential-socket" pod "49b8de5c-62d9-4246-a307-796410ed7bc0" (UID: "49b8de5c-62d9-4246-a307-796410ed7bc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:30:31.822785 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.822766 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-data" pod "49b8de5c-62d9-4246-a307-796410ed7bc0" (UID: "49b8de5c-62d9-4246-a307-796410ed7bc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:30:31.822871 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.822850 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:31.822907 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.822874 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49b8de5c-62d9-4246-a307-796410ed7bc0-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:31.822907 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.822889 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:31.825936 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.825280 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-podinfo" pod "49b8de5c-62d9-4246-a307-796410ed7bc0" (UID: "49b8de5c-62d9-4246-a307-796410ed7bc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:30:31.825936 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.825876 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b8de5c-62d9-4246-a307-796410ed7bc0-kube-api-access-lwp6h" pod "49b8de5c-62d9-4246-a307-796410ed7bc0" (UID: "49b8de5c-62d9-4246-a307-796410ed7bc0"). InnerVolumeSpecName "kube-api-access-lwp6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:31.826122 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.825966 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-envoy" pod "49b8de5c-62d9-4246-a307-796410ed7bc0" (UID: "49b8de5c-62d9-4246-a307-796410ed7bc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:30:31.826563 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.826539 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-token" pod "49b8de5c-62d9-4246-a307-796410ed7bc0" (UID: "49b8de5c-62d9-4246-a307-796410ed7bc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:31.923623 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.923581 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:31.923623 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.923617 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:31.923623 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.923627 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:31.923867 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.923637 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:31.923867 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.923646 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b8de5c-62d9-4246-a307-796410ed7bc0-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:31.923867 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:31.923655 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwp6h\" (UniqueName: \"kubernetes.io/projected/49b8de5c-62d9-4246-a307-796410ed7bc0-kube-api-access-lwp6h\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:32.026914 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:32.026874 5114 patch_prober.go:29] interesting pod/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:30:32.027065 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:32.026945 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" podUID="336646a8-fbe7-4780-85b8-116a012ea309" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:30:32.257337 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:32.257243 5114 generic.go:356] "Generic (PLEG): container finished" podID="49b8de5c-62d9-4246-a307-796410ed7bc0" containerID="1ff52d90e3618a92c6fe8c9a85d49706b5d89b57f3623f9c52ccb6b4ece91c1d" exitCode=0 Nov 02 23:30:32.257337 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:32.257320 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" Nov 02 23:30:32.257866 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:32.257330 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" event={"ID":"49b8de5c-62d9-4246-a307-796410ed7bc0","Type":"ContainerDied","Data":"1ff52d90e3618a92c6fe8c9a85d49706b5d89b57f3623f9c52ccb6b4ece91c1d"} Nov 02 23:30:32.257866 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:32.257386 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx" event={"ID":"49b8de5c-62d9-4246-a307-796410ed7bc0","Type":"ContainerDied","Data":"3424f061151b2947655ad51edb147717d2d50cd7001d38eb7cce7354f4244c97"} Nov 02 23:30:32.257866 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:32.257418 5114 scope.go:122] "RemoveContainer" containerID="1ff52d90e3618a92c6fe8c9a85d49706b5d89b57f3623f9c52ccb6b4ece91c1d" Nov 02 23:30:32.268104 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:32.268076 5114 scope.go:122] "RemoveContainer" containerID="1ff52d90e3618a92c6fe8c9a85d49706b5d89b57f3623f9c52ccb6b4ece91c1d" Nov 02 23:30:32.268430 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:30:32.268407 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ff52d90e3618a92c6fe8c9a85d49706b5d89b57f3623f9c52ccb6b4ece91c1d\": container with ID starting with 1ff52d90e3618a92c6fe8c9a85d49706b5d89b57f3623f9c52ccb6b4ece91c1d not found: ID does not exist" containerID="1ff52d90e3618a92c6fe8c9a85d49706b5d89b57f3623f9c52ccb6b4ece91c1d" Nov 02 23:30:32.268553 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:32.268441 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff52d90e3618a92c6fe8c9a85d49706b5d89b57f3623f9c52ccb6b4ece91c1d"} err="failed to get container status \"1ff52d90e3618a92c6fe8c9a85d49706b5d89b57f3623f9c52ccb6b4ece91c1d\": rpc error: code = NotFound desc = could not find container \"1ff52d90e3618a92c6fe8c9a85d49706b5d89b57f3623f9c52ccb6b4ece91c1d\": container with ID starting with 1ff52d90e3618a92c6fe8c9a85d49706b5d89b57f3623f9c52ccb6b4ece91c1d not found: ID does not exist" Nov 02 23:30:32.401551 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:32.401525 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e8ca-fk898" Nov 02 23:30:32.421987 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:32.421952 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx"] Nov 02 23:30:32.425628 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:32.425594 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-kt8gx"] Nov 02 23:30:32.427996 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:32.427972 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28b616d3-66da-4466-8880-de11323f6ad9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b616d3-66da-4466-8880-de11323f6ad9-maas-api-xks-ca\") pod \"28b616d3-66da-4466-8880-de11323f6ad9\" (UID: \"28b616d3-66da-4466-8880-de11323f6ad9\") " Nov 02 23:30:32.428072 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:32.428055 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28b616d3-66da-4466-8880-de11323f6ad9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b616d3-66da-4466-8880-de11323f6ad9-maas-api-openshift-ca\") pod \"28b616d3-66da-4466-8880-de11323f6ad9\" (UID: \"28b616d3-66da-4466-8880-de11323f6ad9\") " Nov 02 23:30:32.428180 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:32.428166 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b616d3-66da-4466-8880-de11323f6ad9-kube-api-access-6sz5q\" (UniqueName: \"kubernetes.io/projected/28b616d3-66da-4466-8880-de11323f6ad9-kube-api-access-6sz5q\") pod \"28b616d3-66da-4466-8880-de11323f6ad9\" (UID: \"28b616d3-66da-4466-8880-de11323f6ad9\") " Nov 02 23:30:32.428446 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:32.428415 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28b616d3-66da-4466-8880-de11323f6ad9-maas-api-openshift-ca" pod "28b616d3-66da-4466-8880-de11323f6ad9" (UID: "28b616d3-66da-4466-8880-de11323f6ad9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:32.428541 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:32.428515 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b616d3-66da-4466-8880-de11323f6ad9-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:32.430520 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:32.430465 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28b616d3-66da-4466-8880-de11323f6ad9-maas-api-xks-ca" pod "28b616d3-66da-4466-8880-de11323f6ad9" (UID: "28b616d3-66da-4466-8880-de11323f6ad9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:30:32.430751 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:32.430730 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b616d3-66da-4466-8880-de11323f6ad9-kube-api-access-6sz5q" pod "28b616d3-66da-4466-8880-de11323f6ad9" (UID: "28b616d3-66da-4466-8880-de11323f6ad9"). InnerVolumeSpecName "kube-api-access-6sz5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:32.529154 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:32.529054 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sz5q\" (UniqueName: \"kubernetes.io/projected/28b616d3-66da-4466-8880-de11323f6ad9-kube-api-access-6sz5q\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:32.529154 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:32.529096 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b616d3-66da-4466-8880-de11323f6ad9-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:32.793526 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:32.793465 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49b8de5c-62d9-4246-a307-796410ed7bc0" path="/var/lib/kubelet/pods/49b8de5c-62d9-4246-a307-796410ed7bc0/volumes" Nov 02 23:30:33.264917 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.264875 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e8ca-fk898" event={"ID":"28b616d3-66da-4466-8880-de11323f6ad9","Type":"ContainerDied","Data":"2bd66722e94388123f77812f666dfb71c74f6958f22e8142264357e775e4738f"} Nov 02 23:30:33.265384 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.264929 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e8ca-fk898" Nov 02 23:30:33.265384 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.264946 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bd66722e94388123f77812f666dfb71c74f6958f22e8142264357e775e4738f" Nov 02 23:30:33.268314 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.268280 5114 generic.go:356] "Generic (PLEG): container finished" podID="336646a8-fbe7-4780-85b8-116a012ea309" containerID="345384af6035acab85499fd1250cf64f75bc32512ddc8bd04197e4d57ec9b8ac" exitCode=0 Nov 02 23:30:33.268444 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.268315 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" event={"ID":"336646a8-fbe7-4780-85b8-116a012ea309","Type":"ContainerDied","Data":"345384af6035acab85499fd1250cf64f75bc32512ddc8bd04197e4d57ec9b8ac"} Nov 02 23:30:33.268444 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.268400 5114 scope.go:122] "RemoveContainer" containerID="b76bf7f54e6edcf8a6066bf539a38bfb4acb4f8d39747ae5578f4084c022314c" Nov 02 23:30:33.402157 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.402088 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:30:33.438473 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.438435 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-workload-socket\") pod \"336646a8-fbe7-4780-85b8-116a012ea309\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " Nov 02 23:30:33.438668 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.438525 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/336646a8-fbe7-4780-85b8-116a012ea309-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/336646a8-fbe7-4780-85b8-116a012ea309-istiod-ca-cert\") pod \"336646a8-fbe7-4780-85b8-116a012ea309\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " Nov 02 23:30:33.438668 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.438556 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/336646a8-fbe7-4780-85b8-116a012ea309-istio-token\" (UniqueName: \"kubernetes.io/projected/336646a8-fbe7-4780-85b8-116a012ea309-istio-token\") pod \"336646a8-fbe7-4780-85b8-116a012ea309\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " Nov 02 23:30:33.438761 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.438669 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-istio-envoy\") pod \"336646a8-fbe7-4780-85b8-116a012ea309\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " Nov 02 23:30:33.438761 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.438732 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-credential-socket\") pod \"336646a8-fbe7-4780-85b8-116a012ea309\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " Nov 02 23:30:33.438761 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.438736 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-workload-socket" pod "336646a8-fbe7-4780-85b8-116a012ea309" (UID: "336646a8-fbe7-4780-85b8-116a012ea309"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:30:33.438908 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.438760 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-istio-data\") pod \"336646a8-fbe7-4780-85b8-116a012ea309\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " Nov 02 23:30:33.438908 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.438849 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/336646a8-fbe7-4780-85b8-116a012ea309-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/336646a8-fbe7-4780-85b8-116a012ea309-istio-podinfo\") pod \"336646a8-fbe7-4780-85b8-116a012ea309\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " Nov 02 23:30:33.438908 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.438895 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/336646a8-fbe7-4780-85b8-116a012ea309-kube-api-access-v2ff8\" (UniqueName: \"kubernetes.io/projected/336646a8-fbe7-4780-85b8-116a012ea309-kube-api-access-v2ff8\") pod \"336646a8-fbe7-4780-85b8-116a012ea309\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " Nov 02 23:30:33.439073 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.438928 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-workload-certs\") pod \"336646a8-fbe7-4780-85b8-116a012ea309\" (UID: \"336646a8-fbe7-4780-85b8-116a012ea309\") " Nov 02 23:30:33.439073 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.438995 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-credential-socket" pod "336646a8-fbe7-4780-85b8-116a012ea309" (UID: "336646a8-fbe7-4780-85b8-116a012ea309"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:30:33.439073 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.439016 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/336646a8-fbe7-4780-85b8-116a012ea309-istiod-ca-cert" pod "336646a8-fbe7-4780-85b8-116a012ea309" (UID: "336646a8-fbe7-4780-85b8-116a012ea309"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:33.439237 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.439099 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-istio-data" pod "336646a8-fbe7-4780-85b8-116a012ea309" (UID: "336646a8-fbe7-4780-85b8-116a012ea309"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:30:33.439323 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.439300 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-workload-certs" pod "336646a8-fbe7-4780-85b8-116a012ea309" (UID: "336646a8-fbe7-4780-85b8-116a012ea309"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:30:33.439574 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.439520 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:33.439672 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.439581 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:33.439783 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.439708 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:33.439834 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.439794 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:33.439834 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.439809 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/336646a8-fbe7-4780-85b8-116a012ea309-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:33.441209 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.441181 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336646a8-fbe7-4780-85b8-116a012ea309-istio-token" pod "336646a8-fbe7-4780-85b8-116a012ea309" (UID: "336646a8-fbe7-4780-85b8-116a012ea309"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:33.441209 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.441179 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/336646a8-fbe7-4780-85b8-116a012ea309-istio-podinfo" pod "336646a8-fbe7-4780-85b8-116a012ea309" (UID: "336646a8-fbe7-4780-85b8-116a012ea309"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:30:33.441621 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.441597 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-istio-envoy" pod "336646a8-fbe7-4780-85b8-116a012ea309" (UID: "336646a8-fbe7-4780-85b8-116a012ea309"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:30:33.441716 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.441634 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336646a8-fbe7-4780-85b8-116a012ea309-kube-api-access-v2ff8" pod "336646a8-fbe7-4780-85b8-116a012ea309" (UID: "336646a8-fbe7-4780-85b8-116a012ea309"). InnerVolumeSpecName "kube-api-access-v2ff8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:33.551462 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.551420 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/336646a8-fbe7-4780-85b8-116a012ea309-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:33.551462 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.551464 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2ff8\" (UniqueName: \"kubernetes.io/projected/336646a8-fbe7-4780-85b8-116a012ea309-kube-api-access-v2ff8\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:33.551761 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.551499 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/336646a8-fbe7-4780-85b8-116a012ea309-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:33.551761 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:33.551516 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/336646a8-fbe7-4780-85b8-116a012ea309-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:34.276297 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:34.276265 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" Nov 02 23:30:34.276834 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:34.276303 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h" event={"ID":"336646a8-fbe7-4780-85b8-116a012ea309","Type":"ContainerDied","Data":"b6f19d0b627f95296c6bff087b2537c52c45226e040331d8ae29055ac62b1952"} Nov 02 23:30:34.276834 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:34.276366 5114 scope.go:122] "RemoveContainer" containerID="345384af6035acab85499fd1250cf64f75bc32512ddc8bd04197e4d57ec9b8ac" Nov 02 23:30:34.489949 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:34.489914 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h"] Nov 02 23:30:34.539706 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:34.539662 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-85efc6-openshift-default-8475949f45-xf28h"] Nov 02 23:30:34.793710 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:34.793621 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="336646a8-fbe7-4780-85b8-116a012ea309" path="/var/lib/kubelet/pods/336646a8-fbe7-4780-85b8-116a012ea309/volumes" Nov 02 23:30:37.584583 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:30:37.584540 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:30:39.439359 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.439326 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ea7652b-wk5jw"] Nov 02 23:30:39.439959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.439897 5114 cpu_manager.go:414] "containerMap: removing container" podUID="336646a8-fbe7-4780-85b8-116a012ea309" containerName="istio-proxy" Nov 02 23:30:39.439959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.439918 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="336646a8-fbe7-4780-85b8-116a012ea309" containerName="istio-proxy" Nov 02 23:30:39.439959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.439945 5114 cpu_manager.go:414] "containerMap: removing container" podUID="fa2d7e3f-05c4-4628-9f7e-5e463a12aba4" containerName="cleanup" Nov 02 23:30:39.439959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.439953 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2d7e3f-05c4-4628-9f7e-5e463a12aba4" containerName="cleanup" Nov 02 23:30:39.440195 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.439971 5114 cpu_manager.go:414] "containerMap: removing container" podUID="f45d8aa7-3576-4203-bc85-9274ec4d6874" containerName="cleanup" Nov 02 23:30:39.440195 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.439977 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45d8aa7-3576-4203-bc85-9274ec4d6874" containerName="cleanup" Nov 02 23:30:39.440195 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.439988 5114 cpu_manager.go:414] "containerMap: removing container" podUID="49b8de5c-62d9-4246-a307-796410ed7bc0" containerName="istio-proxy" Nov 02 23:30:39.440195 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.439995 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49b8de5c-62d9-4246-a307-796410ed7bc0" containerName="istio-proxy" Nov 02 23:30:39.440195 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.440012 5114 cpu_manager.go:414] "containerMap: removing container" podUID="336646a8-fbe7-4780-85b8-116a012ea309" containerName="istio-proxy" Nov 02 23:30:39.440195 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.440017 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="336646a8-fbe7-4780-85b8-116a012ea309" containerName="istio-proxy" Nov 02 23:30:39.440195 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.440032 5114 cpu_manager.go:414] "containerMap: removing container" podUID="28b616d3-66da-4466-8880-de11323f6ad9" containerName="revoke-keys" Nov 02 23:30:39.440195 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.440037 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28b616d3-66da-4466-8880-de11323f6ad9" containerName="revoke-keys" Nov 02 23:30:39.440195 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.440140 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2d7e3f-05c4-4628-9f7e-5e463a12aba4" containerName="cleanup" Nov 02 23:30:39.440195 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.440152 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="49b8de5c-62d9-4246-a307-796410ed7bc0" containerName="istio-proxy" Nov 02 23:30:39.440195 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.440160 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="880cd41f-4818-4140-bc4d-6f8676e2feff" containerName="cleanup" Nov 02 23:30:39.440195 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.440170 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45d8aa7-3576-4203-bc85-9274ec4d6874" containerName="cleanup" Nov 02 23:30:39.440195 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.440176 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="336646a8-fbe7-4780-85b8-116a012ea309" containerName="istio-proxy" Nov 02 23:30:39.440195 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.440184 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="28b616d3-66da-4466-8880-de11323f6ad9" containerName="revoke-keys" Nov 02 23:30:39.444823 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.444804 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea7652b-wk5jw" Nov 02 23:30:39.498400 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.498363 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:30:39.513959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.513925 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8de33585-6958-4a92-b698-a70b1ec67165-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ea7652b-wk5jw\" (UID: \"8de33585-6958-4a92-b698-a70b1ec67165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea7652b-wk5jw" Nov 02 23:30:39.514174 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.513994 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-867gc\" (UniqueName: \"kubernetes.io/projected/8de33585-6958-4a92-b698-a70b1ec67165-kube-api-access-867gc\") pod \"maas-api-revoke-sub-8ea7652b-wk5jw\" (UID: \"8de33585-6958-4a92-b698-a70b1ec67165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea7652b-wk5jw" Nov 02 23:30:39.514174 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.514100 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8de33585-6958-4a92-b698-a70b1ec67165-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ea7652b-wk5jw\" (UID: \"8de33585-6958-4a92-b698-a70b1ec67165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea7652b-wk5jw" Nov 02 23:30:39.547955 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.547908 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:30:39.550014 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.549979 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ea7652b-wk5jw"] Nov 02 23:30:39.614900 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.614864 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8de33585-6958-4a92-b698-a70b1ec67165-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ea7652b-wk5jw\" (UID: \"8de33585-6958-4a92-b698-a70b1ec67165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea7652b-wk5jw" Nov 02 23:30:39.615111 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.614939 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-867gc\" (UniqueName: \"kubernetes.io/projected/8de33585-6958-4a92-b698-a70b1ec67165-kube-api-access-867gc\") pod \"maas-api-revoke-sub-8ea7652b-wk5jw\" (UID: \"8de33585-6958-4a92-b698-a70b1ec67165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea7652b-wk5jw" Nov 02 23:30:39.615111 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.614961 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8de33585-6958-4a92-b698-a70b1ec67165-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ea7652b-wk5jw\" (UID: \"8de33585-6958-4a92-b698-a70b1ec67165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea7652b-wk5jw" Nov 02 23:30:39.615635 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.615607 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8de33585-6958-4a92-b698-a70b1ec67165-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ea7652b-wk5jw\" (UID: \"8de33585-6958-4a92-b698-a70b1ec67165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea7652b-wk5jw" Nov 02 23:30:39.617448 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.617430 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8de33585-6958-4a92-b698-a70b1ec67165-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ea7652b-wk5jw\" (UID: \"8de33585-6958-4a92-b698-a70b1ec67165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea7652b-wk5jw" Nov 02 23:30:39.676008 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.675968 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-867gc\" (UniqueName: \"kubernetes.io/projected/8de33585-6958-4a92-b698-a70b1ec67165-kube-api-access-867gc\") pod \"maas-api-revoke-sub-8ea7652b-wk5jw\" (UID: \"8de33585-6958-4a92-b698-a70b1ec67165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea7652b-wk5jw" Nov 02 23:30:39.757744 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:39.757655 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea7652b-wk5jw" Nov 02 23:30:40.180164 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:40.180108 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ea7652b-wk5jw"] Nov 02 23:30:40.182691 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:30:40.182660 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de33585_6958_4a92_b698_a70b1ec67165.slice/crio-18506e0828ef824dc13e8096e640bb2a83572741ed3df4c3901c9d3680086bc4 WatchSource:0}: Error finding container 18506e0828ef824dc13e8096e640bb2a83572741ed3df4c3901c9d3680086bc4: Status 404 returned error can't find the container with id 18506e0828ef824dc13e8096e640bb2a83572741ed3df4c3901c9d3680086bc4 Nov 02 23:30:40.312870 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:40.312825 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea7652b-wk5jw" event={"ID":"8de33585-6958-4a92-b698-a70b1ec67165","Type":"ContainerStarted","Data":"18506e0828ef824dc13e8096e640bb2a83572741ed3df4c3901c9d3680086bc4"} Nov 02 23:30:41.319709 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:41.319663 5114 generic.go:356] "Generic (PLEG): container finished" podID="8de33585-6958-4a92-b698-a70b1ec67165" containerID="e46493c075dfd9155b5b21196b9f4eb6f9ba20131cec2c9f7d68308037deb27a" exitCode=0 Nov 02 23:30:41.320157 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:41.319744 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea7652b-wk5jw" event={"ID":"8de33585-6958-4a92-b698-a70b1ec67165","Type":"ContainerDied","Data":"e46493c075dfd9155b5b21196b9f4eb6f9ba20131cec2c9f7d68308037deb27a"} Nov 02 23:30:42.473153 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:42.473125 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea7652b-wk5jw" Nov 02 23:30:42.549865 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:42.549829 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8de33585-6958-4a92-b698-a70b1ec67165-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8de33585-6958-4a92-b698-a70b1ec67165-maas-api-openshift-ca\") pod \"8de33585-6958-4a92-b698-a70b1ec67165\" (UID: \"8de33585-6958-4a92-b698-a70b1ec67165\") " Nov 02 23:30:42.550072 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:42.549905 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8de33585-6958-4a92-b698-a70b1ec67165-kube-api-access-867gc\" (UniqueName: \"kubernetes.io/projected/8de33585-6958-4a92-b698-a70b1ec67165-kube-api-access-867gc\") pod \"8de33585-6958-4a92-b698-a70b1ec67165\" (UID: \"8de33585-6958-4a92-b698-a70b1ec67165\") " Nov 02 23:30:42.550072 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:42.549954 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8de33585-6958-4a92-b698-a70b1ec67165-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8de33585-6958-4a92-b698-a70b1ec67165-maas-api-xks-ca\") pod \"8de33585-6958-4a92-b698-a70b1ec67165\" (UID: \"8de33585-6958-4a92-b698-a70b1ec67165\") " Nov 02 23:30:42.550275 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:42.550248 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8de33585-6958-4a92-b698-a70b1ec67165-maas-api-openshift-ca" pod "8de33585-6958-4a92-b698-a70b1ec67165" (UID: "8de33585-6958-4a92-b698-a70b1ec67165"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:42.552192 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:42.552159 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8de33585-6958-4a92-b698-a70b1ec67165-maas-api-xks-ca" pod "8de33585-6958-4a92-b698-a70b1ec67165" (UID: "8de33585-6958-4a92-b698-a70b1ec67165"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:30:42.552315 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:42.552225 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de33585-6958-4a92-b698-a70b1ec67165-kube-api-access-867gc" pod "8de33585-6958-4a92-b698-a70b1ec67165" (UID: "8de33585-6958-4a92-b698-a70b1ec67165"). InnerVolumeSpecName "kube-api-access-867gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:42.651730 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:42.651642 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8de33585-6958-4a92-b698-a70b1ec67165-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:42.651730 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:42.651680 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8de33585-6958-4a92-b698-a70b1ec67165-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:42.651730 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:42.651693 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-867gc\" (UniqueName: \"kubernetes.io/projected/8de33585-6958-4a92-b698-a70b1ec67165-kube-api-access-867gc\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:43.333004 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:43.332952 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea7652b-wk5jw" event={"ID":"8de33585-6958-4a92-b698-a70b1ec67165","Type":"ContainerDied","Data":"18506e0828ef824dc13e8096e640bb2a83572741ed3df4c3901c9d3680086bc4"} Nov 02 23:30:43.333004 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:43.332990 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea7652b-wk5jw" Nov 02 23:30:43.333253 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:43.333026 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18506e0828ef824dc13e8096e640bb2a83572741ed3df4c3901c9d3680086bc4" Nov 02 23:30:44.312568 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:44.312500 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" podUID="6e3d93da-6d85-4268-9261-403c456aa765" containerName="main" containerID="cri-o://55fc1d02d62a701eca53449e1b27d85bad933d7029f2448cdd0e8ab0e7d9f78b" gracePeriod=45 Nov 02 23:30:44.577202 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:44.577133 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:30:44.673564 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:44.673530 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-home\" (UniqueName: \"kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-home\") pod \"6e3d93da-6d85-4268-9261-403c456aa765\" (UID: \"6e3d93da-6d85-4268-9261-403c456aa765\") " Nov 02 23:30:44.673772 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:44.673600 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e3d93da-6d85-4268-9261-403c456aa765-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e3d93da-6d85-4268-9261-403c456aa765-tls-certs\") pod \"6e3d93da-6d85-4268-9261-403c456aa765\" (UID: \"6e3d93da-6d85-4268-9261-403c456aa765\") " Nov 02 23:30:44.673772 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:44.673638 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-model-cache\") pod \"6e3d93da-6d85-4268-9261-403c456aa765\" (UID: \"6e3d93da-6d85-4268-9261-403c456aa765\") " Nov 02 23:30:44.673772 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:44.673680 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e3d93da-6d85-4268-9261-403c456aa765-kube-api-access-545qz\" (UniqueName: \"kubernetes.io/projected/6e3d93da-6d85-4268-9261-403c456aa765-kube-api-access-545qz\") pod \"6e3d93da-6d85-4268-9261-403c456aa765\" (UID: \"6e3d93da-6d85-4268-9261-403c456aa765\") " Nov 02 23:30:44.673772 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:44.673738 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-dshm\") pod \"6e3d93da-6d85-4268-9261-403c456aa765\" (UID: \"6e3d93da-6d85-4268-9261-403c456aa765\") " Nov 02 23:30:44.673996 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:44.673847 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-tmp-dir\") pod \"6e3d93da-6d85-4268-9261-403c456aa765\" (UID: \"6e3d93da-6d85-4268-9261-403c456aa765\") " Nov 02 23:30:44.673996 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:44.673889 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-home" pod "6e3d93da-6d85-4268-9261-403c456aa765" (UID: "6e3d93da-6d85-4268-9261-403c456aa765"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:30:44.673996 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:44.673987 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-model-cache" pod "6e3d93da-6d85-4268-9261-403c456aa765" (UID: "6e3d93da-6d85-4268-9261-403c456aa765"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:30:44.674139 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:44.674106 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-tmp-dir" pod "6e3d93da-6d85-4268-9261-403c456aa765" (UID: "6e3d93da-6d85-4268-9261-403c456aa765"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:30:44.674231 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:44.674215 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:44.674283 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:44.674233 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:44.674283 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:44.674242 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:44.676346 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:44.676316 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e3d93da-6d85-4268-9261-403c456aa765-tls-certs" pod "6e3d93da-6d85-4268-9261-403c456aa765" (UID: "6e3d93da-6d85-4268-9261-403c456aa765"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:30:44.676589 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:44.676566 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e3d93da-6d85-4268-9261-403c456aa765-kube-api-access-545qz" pod "6e3d93da-6d85-4268-9261-403c456aa765" (UID: "6e3d93da-6d85-4268-9261-403c456aa765"). InnerVolumeSpecName "kube-api-access-545qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:44.676694 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:44.676582 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-dshm" pod "6e3d93da-6d85-4268-9261-403c456aa765" (UID: "6e3d93da-6d85-4268-9261-403c456aa765"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:30:44.775236 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:44.775189 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e3d93da-6d85-4268-9261-403c456aa765-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:44.775236 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:44.775231 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-545qz\" (UniqueName: \"kubernetes.io/projected/6e3d93da-6d85-4268-9261-403c456aa765-kube-api-access-545qz\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:44.775236 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:44.775241 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e3d93da-6d85-4268-9261-403c456aa765-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:45.344873 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:45.344845 5114 generic.go:356] "Generic (PLEG): container finished" podID="6e3d93da-6d85-4268-9261-403c456aa765" containerID="55fc1d02d62a701eca53449e1b27d85bad933d7029f2448cdd0e8ab0e7d9f78b" exitCode=0 Nov 02 23:30:45.345292 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:45.344922 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" Nov 02 23:30:45.345292 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:45.344935 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" event={"ID":"6e3d93da-6d85-4268-9261-403c456aa765","Type":"ContainerDied","Data":"55fc1d02d62a701eca53449e1b27d85bad933d7029f2448cdd0e8ab0e7d9f78b"} Nov 02 23:30:45.345292 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:45.344988 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7" event={"ID":"6e3d93da-6d85-4268-9261-403c456aa765","Type":"ContainerDied","Data":"b3d4752989cdc6981461d11d09a1c357c6375f220d50d9ccda26a7c8813b3a29"} Nov 02 23:30:45.345292 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:45.345008 5114 scope.go:122] "RemoveContainer" containerID="55fc1d02d62a701eca53449e1b27d85bad933d7029f2448cdd0e8ab0e7d9f78b" Nov 02 23:30:45.355691 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:45.355668 5114 scope.go:122] "RemoveContainer" containerID="55fc1d02d62a701eca53449e1b27d85bad933d7029f2448cdd0e8ab0e7d9f78b" Nov 02 23:30:45.355998 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:30:45.355965 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55fc1d02d62a701eca53449e1b27d85bad933d7029f2448cdd0e8ab0e7d9f78b\": container with ID starting with 55fc1d02d62a701eca53449e1b27d85bad933d7029f2448cdd0e8ab0e7d9f78b not found: ID does not exist" containerID="55fc1d02d62a701eca53449e1b27d85bad933d7029f2448cdd0e8ab0e7d9f78b" Nov 02 23:30:45.356050 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:45.356003 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55fc1d02d62a701eca53449e1b27d85bad933d7029f2448cdd0e8ab0e7d9f78b"} err="failed to get container status \"55fc1d02d62a701eca53449e1b27d85bad933d7029f2448cdd0e8ab0e7d9f78b\": rpc error: code = NotFound desc = could not find container \"55fc1d02d62a701eca53449e1b27d85bad933d7029f2448cdd0e8ab0e7d9f78b\": container with ID starting with 55fc1d02d62a701eca53449e1b27d85bad933d7029f2448cdd0e8ab0e7d9f78b not found: ID does not exist" Nov 02 23:30:45.509589 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:45.509526 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7"] Nov 02 23:30:45.512448 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:45.512415 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-1cb1f8/ipp-route-1cb1f8-kserve-959869577-2lsd7"] Nov 02 23:30:46.793737 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:46.793703 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e3d93da-6d85-4268-9261-403c456aa765" path="/var/lib/kubelet/pods/6e3d93da-6d85-4268-9261-403c456aa765/volumes" Nov 02 23:30:50.319221 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:50.319180 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s"] Nov 02 23:30:50.319754 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:50.319585 5114 cpu_manager.go:414] "containerMap: removing container" podUID="8de33585-6958-4a92-b698-a70b1ec67165" containerName="revoke-keys" Nov 02 23:30:50.319754 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:50.319601 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8de33585-6958-4a92-b698-a70b1ec67165" containerName="revoke-keys" Nov 02 23:30:50.319754 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:50.319621 5114 cpu_manager.go:414] "containerMap: removing container" podUID="6e3d93da-6d85-4268-9261-403c456aa765" containerName="main" Nov 02 23:30:50.319754 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:50.319626 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e3d93da-6d85-4268-9261-403c456aa765" containerName="main" Nov 02 23:30:50.319754 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:50.319711 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e3d93da-6d85-4268-9261-403c456aa765" containerName="main" Nov 02 23:30:50.319754 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:50.319727 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="336646a8-fbe7-4780-85b8-116a012ea309" containerName="istio-proxy" Nov 02 23:30:50.319754 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:50.319740 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="8de33585-6958-4a92-b698-a70b1ec67165" containerName="revoke-keys" Nov 02 23:30:50.324584 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:50.324565 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s" Nov 02 23:30:50.381280 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:50.381244 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s"] Nov 02 23:30:50.428953 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:50.428916 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-818fe8\"" Nov 02 23:30:50.428953 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:50.428951 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-818fe8-dockercfg-znrgv\"" Nov 02 23:30:50.437696 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:50.437669 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bczh\" (UniqueName: \"kubernetes.io/projected/8bd9d853-a244-4206-b198-24652ae68088-kube-api-access-4bczh\") pod \"payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s\" (UID: \"8bd9d853-a244-4206-b198-24652ae68088\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s" Nov 02 23:30:50.437776 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:50.437717 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd9d853-a244-4206-b198-24652ae68088-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s\" (UID: \"8bd9d853-a244-4206-b198-24652ae68088\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s" Nov 02 23:30:50.539233 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:50.539197 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bczh\" (UniqueName: \"kubernetes.io/projected/8bd9d853-a244-4206-b198-24652ae68088-kube-api-access-4bczh\") pod \"payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s\" (UID: \"8bd9d853-a244-4206-b198-24652ae68088\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s" Nov 02 23:30:50.539444 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:50.539249 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd9d853-a244-4206-b198-24652ae68088-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s\" (UID: \"8bd9d853-a244-4206-b198-24652ae68088\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s" Nov 02 23:30:50.540050 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:50.540029 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd9d853-a244-4206-b198-24652ae68088-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s\" (UID: \"8bd9d853-a244-4206-b198-24652ae68088\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s" Nov 02 23:30:50.600528 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:50.600423 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bczh\" (UniqueName: \"kubernetes.io/projected/8bd9d853-a244-4206-b198-24652ae68088-kube-api-access-4bczh\") pod \"payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s\" (UID: \"8bd9d853-a244-4206-b198-24652ae68088\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s" Nov 02 23:30:50.638310 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:50.638273 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s" Nov 02 23:30:51.027184 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:51.027154 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s"] Nov 02 23:30:51.028957 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:30:51.028921 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd9d853_a244_4206_b198_24652ae68088.slice/crio-a2836beb05281ff1e29809bf4db56274814d63ac32ec87b7f84b5757bf22a29d WatchSource:0}: Error finding container a2836beb05281ff1e29809bf4db56274814d63ac32ec87b7f84b5757bf22a29d: Status 404 returned error can't find the container with id a2836beb05281ff1e29809bf4db56274814d63ac32ec87b7f84b5757bf22a29d Nov 02 23:30:51.379333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:51.379291 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s" event={"ID":"8bd9d853-a244-4206-b198-24652ae68088","Type":"ContainerStarted","Data":"a2836beb05281ff1e29809bf4db56274814d63ac32ec87b7f84b5757bf22a29d"} Nov 02 23:30:53.392633 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:53.392598 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s" event={"ID":"8bd9d853-a244-4206-b198-24652ae68088","Type":"ContainerStarted","Data":"b00a6960d04d8049ac760168b4f1db8d0ab9eb2743932cf342f9a6e5958fd541"} Nov 02 23:30:53.393111 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:53.392732 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s" Nov 02 23:30:53.509347 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:53.509289 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s" podStartSLOduration=1.273443253 podStartE2EDuration="3.509271462s" totalImagesPullingTime="2.235828209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:53.507644933 +0000 UTC m=+2528.860678308" watchObservedRunningTime="2026-11-02 23:30:53.509271462 +0000 UTC m=+2528.862304833" Nov 02 23:30:56.234010 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.233963 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx"] Nov 02 23:30:56.239270 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.239078 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.292732 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.292694 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.292936 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.292741 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.292936 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.292847 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.292936 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.292876 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.292936 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.292897 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0003757b-3035-4a01-9aac-0f9f10c0c845-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.293196 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.292970 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.293196 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.293017 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.293196 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.293102 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqtdc\" (UniqueName: \"kubernetes.io/projected/0003757b-3035-4a01-9aac-0f9f10c0c845-kube-api-access-hqtdc\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.293196 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.293146 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.343325 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.343277 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx"] Nov 02 23:30:56.393760 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.393720 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.393760 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.393773 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.394043 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.393813 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.394043 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.393828 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.394043 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.393844 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0003757b-3035-4a01-9aac-0f9f10c0c845-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.394043 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.393866 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.394043 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.393893 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.394043 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.393949 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqtdc\" (UniqueName: \"kubernetes.io/projected/0003757b-3035-4a01-9aac-0f9f10c0c845-kube-api-access-hqtdc\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.394043 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.393997 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.394605 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.394308 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.394605 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.394353 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.394605 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.394429 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.394605 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.394594 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.394809 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.394715 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0003757b-3035-4a01-9aac-0f9f10c0c845-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.396467 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.396440 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.396764 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.396747 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.454877 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.454839 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.455054 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.454889 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqtdc\" (UniqueName: \"kubernetes.io/projected/0003757b-3035-4a01-9aac-0f9f10c0c845-kube-api-access-hqtdc\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.555340 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.555299 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:56.988089 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.988061 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx"] Nov 02 23:30:56.990147 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:30:56.990122 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0003757b_3035_4a01_9aac_0f9f10c0c845.slice/crio-43ab9d8c11c235f8cb9012e95a5b743b33eb253efbd17346ef426b8aaade9085 WatchSource:0}: Error finding container 43ab9d8c11c235f8cb9012e95a5b743b33eb253efbd17346ef426b8aaade9085: Status 404 returned error can't find the container with id 43ab9d8c11c235f8cb9012e95a5b743b33eb253efbd17346ef426b8aaade9085 Nov 02 23:30:56.992816 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.992787 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:30:56.992887 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.992876 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:30:56.992925 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:56.992903 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:30:57.417649 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:57.417601 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" event={"ID":"0003757b-3035-4a01-9aac-0f9f10c0c845","Type":"ContainerStarted","Data":"bf4c7ef11e8fcd50c4cee8a0e517172355b695f75fe4844932787552a8e937d8"} Nov 02 23:30:57.418339 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:57.417672 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" event={"ID":"0003757b-3035-4a01-9aac-0f9f10c0c845","Type":"ContainerStarted","Data":"43ab9d8c11c235f8cb9012e95a5b743b33eb253efbd17346ef426b8aaade9085"} Nov 02 23:30:57.555878 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:57.555840 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:30:58.559128 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:58.559088 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:30:58.559566 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:58.559153 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" podUID="0003757b-3035-4a01-9aac-0f9f10c0c845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 23:30:59.556417 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:59.556371 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:30:59.556647 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:30:59.556440 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" podUID="0003757b-3035-4a01-9aac-0f9f10c0c845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 23:31:00.556415 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:00.556378 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:31:00.556864 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:00.556432 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" podUID="0003757b-3035-4a01-9aac-0f9f10c0c845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 23:31:01.555810 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:01.555772 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:31:01.555993 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:01.555854 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" podUID="0003757b-3035-4a01-9aac-0f9f10c0c845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 23:31:02.556683 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:02.556645 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:31:02.557131 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:02.556703 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" podUID="0003757b-3035-4a01-9aac-0f9f10c0c845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 23:31:03.556271 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:03.556233 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:31:03.556450 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:03.556292 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" podUID="0003757b-3035-4a01-9aac-0f9f10c0c845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 23:31:04.401956 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:04.401924 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s" Nov 02 23:31:04.520424 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:04.520344 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" podStartSLOduration=8.520321575 podStartE2EDuration="8.520321575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:57.539819623 +0000 UTC m=+2532.892853010" watchObservedRunningTime="2026-11-02 23:31:04.520321575 +0000 UTC m=+2539.873354951" Nov 02 23:31:04.583811 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:04.583780 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:31:04.584386 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:04.584352 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:31:04.585389 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:04.585370 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:31:07.583946 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:31:07.583906 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:31:26.470043 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.470007 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr"] Nov 02 23:31:26.474273 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.474252 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.526538 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.526505 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ec2f9a-openshift-default-dockercfg-4nt5j\"" Nov 02 23:31:26.540117 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.540073 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t86tn\" (UniqueName: \"kubernetes.io/projected/191ad0c4-d290-448b-99ec-9304cd39778c-kube-api-access-t86tn\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.540302 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.540139 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191ad0c4-d290-448b-99ec-9304cd39778c-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.540302 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.540259 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191ad0c4-d290-448b-99ec-9304cd39778c-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.540404 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.540331 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/191ad0c4-d290-448b-99ec-9304cd39778c-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.540404 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.540373 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.540404 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.540393 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.540573 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.540462 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.540573 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.540505 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.540573 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.540529 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.578989 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.578950 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr"] Nov 02 23:31:26.641687 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.641650 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.641687 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.641689 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.641962 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.641709 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.641962 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.641768 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t86tn\" (UniqueName: \"kubernetes.io/projected/191ad0c4-d290-448b-99ec-9304cd39778c-kube-api-access-t86tn\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.641962 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.641803 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191ad0c4-d290-448b-99ec-9304cd39778c-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.641962 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.641865 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191ad0c4-d290-448b-99ec-9304cd39778c-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.641962 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.641916 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/191ad0c4-d290-448b-99ec-9304cd39778c-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.641962 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.641942 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.642287 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.641966 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.642287 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.642113 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.642287 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.642265 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.642403 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.642287 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.642533 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.642474 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.642622 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.642603 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191ad0c4-d290-448b-99ec-9304cd39778c-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.644359 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.644333 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.644653 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.644636 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191ad0c4-d290-448b-99ec-9304cd39778c-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.702698 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.702652 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/191ad0c4-d290-448b-99ec-9304cd39778c-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.702833 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.702759 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t86tn\" (UniqueName: \"kubernetes.io/projected/191ad0c4-d290-448b-99ec-9304cd39778c-kube-api-access-t86tn\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:26.788184 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:26.788147 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:27.217007 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:27.216978 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr"] Nov 02 23:31:27.219612 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:31:27.219580 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191ad0c4_d290_448b_99ec_9304cd39778c.slice/crio-b2ac5e1c10e31d3ba3953194e40e3acdc886d4ec0ea06970541959084e84fcb7 WatchSource:0}: Error finding container b2ac5e1c10e31d3ba3953194e40e3acdc886d4ec0ea06970541959084e84fcb7: Status 404 returned error can't find the container with id b2ac5e1c10e31d3ba3953194e40e3acdc886d4ec0ea06970541959084e84fcb7 Nov 02 23:31:27.222346 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:27.222310 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:31:27.222453 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:27.222403 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:31:27.222453 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:27.222445 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:31:27.591838 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:27.591795 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" event={"ID":"191ad0c4-d290-448b-99ec-9304cd39778c","Type":"ContainerStarted","Data":"657ab192bb55bfa4e7adcf59baafa53da6d9e2f18e8f6eb04329682a35d97237"} Nov 02 23:31:27.592375 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:27.591907 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" event={"ID":"191ad0c4-d290-448b-99ec-9304cd39778c","Type":"ContainerStarted","Data":"b2ac5e1c10e31d3ba3953194e40e3acdc886d4ec0ea06970541959084e84fcb7"} Nov 02 23:31:27.713142 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:27.713075 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" podStartSLOduration=1.7130573230000001 podStartE2EDuration="1.713057323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:31:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:31:27.711044031 +0000 UTC m=+2563.064077406" watchObservedRunningTime="2026-11-02 23:31:27.713057323 +0000 UTC m=+2563.066090699" Nov 02 23:31:27.789052 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:27.789009 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:28.790566 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:28.790529 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 23:31:28.791066 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:28.790608 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" podUID="191ad0c4-d290-448b-99ec-9304cd39778c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 23:31:29.789591 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:29.789559 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 23:31:29.789746 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:29.789623 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" podUID="191ad0c4-d290-448b-99ec-9304cd39778c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 23:31:30.788581 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:30.788547 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 23:31:30.789000 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:30.788612 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" podUID="191ad0c4-d290-448b-99ec-9304cd39778c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 23:31:31.788568 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:31.788535 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 23:31:31.788760 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:31.788589 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" podUID="191ad0c4-d290-448b-99ec-9304cd39778c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 23:31:32.789032 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:32.788993 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 23:31:32.789607 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:32.789096 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" podUID="191ad0c4-d290-448b-99ec-9304cd39778c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 23:31:33.788717 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:33.788676 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 23:31:33.788908 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:33.788745 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" podUID="191ad0c4-d290-448b-99ec-9304cd39778c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 23:31:34.796277 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:34.796242 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:34.796813 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:34.796689 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:34.797590 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:34.797565 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:31:37.584749 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:31:37.584709 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:31:54.024858 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:54.024813 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj"] Nov 02 23:31:54.025731 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:54.025079 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" podUID="3f88718e-12ac-41f3-a041-ad7a676d6e0e" containerName="maas-api" containerID="cri-o://04ea09333c2c7e00f2e4a2907b74514e2452310f13107aefcbabcfa874299209" gracePeriod=30 Nov 02 23:31:54.324095 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:54.324061 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" Nov 02 23:31:54.399722 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:54.399691 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f88718e-12ac-41f3-a041-ad7a676d6e0e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f88718e-12ac-41f3-a041-ad7a676d6e0e-metrics-certs\") pod \"3f88718e-12ac-41f3-a041-ad7a676d6e0e\" (UID: \"3f88718e-12ac-41f3-a041-ad7a676d6e0e\") " Nov 02 23:31:54.399916 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:54.399731 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f88718e-12ac-41f3-a041-ad7a676d6e0e-kube-api-access-fr62k\" (UniqueName: \"kubernetes.io/projected/3f88718e-12ac-41f3-a041-ad7a676d6e0e-kube-api-access-fr62k\") pod \"3f88718e-12ac-41f3-a041-ad7a676d6e0e\" (UID: \"3f88718e-12ac-41f3-a041-ad7a676d6e0e\") " Nov 02 23:31:54.399916 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:54.399794 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f88718e-12ac-41f3-a041-ad7a676d6e0e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f88718e-12ac-41f3-a041-ad7a676d6e0e-openshift-service-ca\") pod \"3f88718e-12ac-41f3-a041-ad7a676d6e0e\" (UID: \"3f88718e-12ac-41f3-a041-ad7a676d6e0e\") " Nov 02 23:31:54.399916 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:54.399849 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f88718e-12ac-41f3-a041-ad7a676d6e0e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f88718e-12ac-41f3-a041-ad7a676d6e0e-maas-api-tls\") pod \"3f88718e-12ac-41f3-a041-ad7a676d6e0e\" (UID: \"3f88718e-12ac-41f3-a041-ad7a676d6e0e\") " Nov 02 23:31:54.400190 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:54.400163 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f88718e-12ac-41f3-a041-ad7a676d6e0e-openshift-service-ca" pod "3f88718e-12ac-41f3-a041-ad7a676d6e0e" (UID: "3f88718e-12ac-41f3-a041-ad7a676d6e0e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:31:54.402276 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:54.402246 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f88718e-12ac-41f3-a041-ad7a676d6e0e-kube-api-access-fr62k" pod "3f88718e-12ac-41f3-a041-ad7a676d6e0e" (UID: "3f88718e-12ac-41f3-a041-ad7a676d6e0e"). InnerVolumeSpecName "kube-api-access-fr62k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:31:54.402415 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:54.402375 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f88718e-12ac-41f3-a041-ad7a676d6e0e-maas-api-tls" pod "3f88718e-12ac-41f3-a041-ad7a676d6e0e" (UID: "3f88718e-12ac-41f3-a041-ad7a676d6e0e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:31:54.402797 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:54.402777 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f88718e-12ac-41f3-a041-ad7a676d6e0e-metrics-certs" pod "3f88718e-12ac-41f3-a041-ad7a676d6e0e" (UID: "3f88718e-12ac-41f3-a041-ad7a676d6e0e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:31:54.501501 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:54.501448 5114 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f88718e-12ac-41f3-a041-ad7a676d6e0e-metrics-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:54.501723 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:54.501518 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr62k\" (UniqueName: \"kubernetes.io/projected/3f88718e-12ac-41f3-a041-ad7a676d6e0e-kube-api-access-fr62k\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:54.501723 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:54.501529 5114 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f88718e-12ac-41f3-a041-ad7a676d6e0e-openshift-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:54.501723 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:54.501538 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f88718e-12ac-41f3-a041-ad7a676d6e0e-maas-api-tls\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:54.775726 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:54.775635 5114 generic.go:356] "Generic (PLEG): container finished" podID="3f88718e-12ac-41f3-a041-ad7a676d6e0e" containerID="04ea09333c2c7e00f2e4a2907b74514e2452310f13107aefcbabcfa874299209" exitCode=0 Nov 02 23:31:54.775726 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:54.775690 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" Nov 02 23:31:54.775726 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:54.775715 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" event={"ID":"3f88718e-12ac-41f3-a041-ad7a676d6e0e","Type":"ContainerDied","Data":"04ea09333c2c7e00f2e4a2907b74514e2452310f13107aefcbabcfa874299209"} Nov 02 23:31:54.776021 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:54.775769 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj" event={"ID":"3f88718e-12ac-41f3-a041-ad7a676d6e0e","Type":"ContainerDied","Data":"7c1f6f1a9729ab144ebd48f037eb42e55b84f1c4fe8902b7a839560a424368d6"} Nov 02 23:31:54.776021 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:54.775791 5114 scope.go:122] "RemoveContainer" containerID="04ea09333c2c7e00f2e4a2907b74514e2452310f13107aefcbabcfa874299209" Nov 02 23:31:54.786618 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:54.786595 5114 scope.go:122] "RemoveContainer" containerID="04ea09333c2c7e00f2e4a2907b74514e2452310f13107aefcbabcfa874299209" Nov 02 23:31:54.786967 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:31:54.786947 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04ea09333c2c7e00f2e4a2907b74514e2452310f13107aefcbabcfa874299209\": container with ID starting with 04ea09333c2c7e00f2e4a2907b74514e2452310f13107aefcbabcfa874299209 not found: ID does not exist" containerID="04ea09333c2c7e00f2e4a2907b74514e2452310f13107aefcbabcfa874299209" Nov 02 23:31:54.787014 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:54.786980 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ea09333c2c7e00f2e4a2907b74514e2452310f13107aefcbabcfa874299209"} err="failed to get container status \"04ea09333c2c7e00f2e4a2907b74514e2452310f13107aefcbabcfa874299209\": rpc error: code = NotFound desc = could not find container \"04ea09333c2c7e00f2e4a2907b74514e2452310f13107aefcbabcfa874299209\": container with ID starting with 04ea09333c2c7e00f2e4a2907b74514e2452310f13107aefcbabcfa874299209 not found: ID does not exist" Nov 02 23:31:54.940569 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:54.940527 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj"] Nov 02 23:31:54.990094 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:54.990060 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-adfeab-5f7c46747d-wmjnj"] Nov 02 23:31:56.251640 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.251602 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb"] Nov 02 23:31:56.252287 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.252260 5114 cpu_manager.go:414] "containerMap: removing container" podUID="3f88718e-12ac-41f3-a041-ad7a676d6e0e" containerName="maas-api" Nov 02 23:31:56.252287 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.252285 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f88718e-12ac-41f3-a041-ad7a676d6e0e" containerName="maas-api" Nov 02 23:31:56.252500 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.252382 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f88718e-12ac-41f3-a041-ad7a676d6e0e" containerName="maas-api" Nov 02 23:31:56.257232 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.257207 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.321731 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.321694 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.321731 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.321732 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.321967 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.321755 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd95eaa-10d3-4928-bf81-adcd1981f259-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.321967 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.321862 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.321967 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.321910 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6ztx\" (UniqueName: \"kubernetes.io/projected/7bd95eaa-10d3-4928-bf81-adcd1981f259-kube-api-access-v6ztx\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.322071 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.322000 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.322071 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.322029 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.322071 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.322050 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.322166 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.322157 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.423716 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.423672 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.423934 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.423795 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.423934 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.423825 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.423934 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.423852 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd95eaa-10d3-4928-bf81-adcd1981f259-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.423934 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.423880 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.423934 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.423922 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6ztx\" (UniqueName: \"kubernetes.io/projected/7bd95eaa-10d3-4928-bf81-adcd1981f259-kube-api-access-v6ztx\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.424229 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.423978 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.424229 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.424009 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.424229 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.424037 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.424583 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.424557 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.425280 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.425084 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.425280 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.425234 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.426321 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.426300 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.426704 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.426641 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd95eaa-10d3-4928-bf81-adcd1981f259-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.428708 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.428689 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.455504 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.453828 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.464338 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.464299 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb"] Nov 02 23:31:56.489342 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.489303 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.490059 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.490031 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6ztx\" (UniqueName: \"kubernetes.io/projected/7bd95eaa-10d3-4928-bf81-adcd1981f259-kube-api-access-v6ztx\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.570668 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.570616 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:56.800956 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:56.800920 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f88718e-12ac-41f3-a041-ad7a676d6e0e" path="/var/lib/kubelet/pods/3f88718e-12ac-41f3-a041-ad7a676d6e0e/volumes" Nov 02 23:31:57.013749 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:57.013715 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb"] Nov 02 23:31:57.015682 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:31:57.015646 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd95eaa_10d3_4928_bf81_adcd1981f259.slice/crio-e926874390dd8327e067caf876ccacb988cf015245c8bbc2b2fe9103c3872f2c WatchSource:0}: Error finding container e926874390dd8327e067caf876ccacb988cf015245c8bbc2b2fe9103c3872f2c: Status 404 returned error can't find the container with id e926874390dd8327e067caf876ccacb988cf015245c8bbc2b2fe9103c3872f2c Nov 02 23:31:57.018523 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:57.018454 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:31:57.018614 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:57.018551 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:31:57.018614 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:57.018583 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:31:57.794559 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:57.794522 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" event={"ID":"7bd95eaa-10d3-4928-bf81-adcd1981f259","Type":"ContainerStarted","Data":"85aee179346b724fab77c7db797099ff9ea2790a132a7ac53bee3cd267ec5200"} Nov 02 23:31:57.794993 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:57.794579 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" event={"ID":"7bd95eaa-10d3-4928-bf81-adcd1981f259","Type":"ContainerStarted","Data":"e926874390dd8327e067caf876ccacb988cf015245c8bbc2b2fe9103c3872f2c"} Nov 02 23:31:57.965072 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:57.965007 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" podStartSLOduration=1.9649914320000001 podStartE2EDuration="1.964991432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:31:57.963855867 +0000 UTC m=+2593.316889268" watchObservedRunningTime="2026-11-02 23:31:57.964991432 +0000 UTC m=+2593.318024860" Nov 02 23:31:58.571261 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:58.571224 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:31:58.573082 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:58.573049 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 23:31:58.573242 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:58.573112 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" podUID="7bd95eaa-10d3-4928-bf81-adcd1981f259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 23:31:59.571754 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:59.571718 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 23:31:59.572266 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:31:59.571782 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" podUID="7bd95eaa-10d3-4928-bf81-adcd1981f259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 23:32:00.571846 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:00.571792 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 23:32:00.572267 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:00.571898 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" podUID="7bd95eaa-10d3-4928-bf81-adcd1981f259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 23:32:01.571203 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:01.571161 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 23:32:01.571398 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:01.571238 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" podUID="7bd95eaa-10d3-4928-bf81-adcd1981f259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 23:32:02.571452 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:02.571418 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 23:32:02.571997 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:02.571661 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" podUID="7bd95eaa-10d3-4928-bf81-adcd1981f259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 23:32:03.571640 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:03.571601 5114 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 23:32:03.572088 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:03.571666 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" podUID="7bd95eaa-10d3-4928-bf81-adcd1981f259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 23:32:04.575797 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:04.575767 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:32:04.576205 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:04.576042 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:32:04.576842 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:04.576825 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:32:07.747511 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:32:07.744707 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:32:21.284243 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:21.284204 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51110ea5-v7sdp"] Nov 02 23:32:21.287837 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:21.287817 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51110ea5-v7sdp" Nov 02 23:32:21.342687 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:21.342650 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:32:21.343045 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:21.343005 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f296a3b6-d7ef-477a-8221-b24528a9b8f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51110ea5-v7sdp\" (UID: \"f296a3b6-d7ef-477a-8221-b24528a9b8f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51110ea5-v7sdp" Nov 02 23:32:21.343188 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:21.343171 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f296a3b6-d7ef-477a-8221-b24528a9b8f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51110ea5-v7sdp\" (UID: \"f296a3b6-d7ef-477a-8221-b24528a9b8f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51110ea5-v7sdp" Nov 02 23:32:21.343353 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:21.343317 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7s7p\" (UniqueName: \"kubernetes.io/projected/f296a3b6-d7ef-477a-8221-b24528a9b8f4-kube-api-access-s7s7p\") pod \"maas-api-revoke-sub-51110ea5-v7sdp\" (UID: \"f296a3b6-d7ef-477a-8221-b24528a9b8f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51110ea5-v7sdp" Nov 02 23:32:21.344434 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:21.344410 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51110ea5-v7sdp"] Nov 02 23:32:21.362914 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:21.362878 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:32:21.444362 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:21.444320 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f296a3b6-d7ef-477a-8221-b24528a9b8f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51110ea5-v7sdp\" (UID: \"f296a3b6-d7ef-477a-8221-b24528a9b8f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51110ea5-v7sdp" Nov 02 23:32:21.444616 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:21.444393 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7s7p\" (UniqueName: \"kubernetes.io/projected/f296a3b6-d7ef-477a-8221-b24528a9b8f4-kube-api-access-s7s7p\") pod \"maas-api-revoke-sub-51110ea5-v7sdp\" (UID: \"f296a3b6-d7ef-477a-8221-b24528a9b8f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51110ea5-v7sdp" Nov 02 23:32:21.444616 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:21.444440 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f296a3b6-d7ef-477a-8221-b24528a9b8f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51110ea5-v7sdp\" (UID: \"f296a3b6-d7ef-477a-8221-b24528a9b8f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51110ea5-v7sdp" Nov 02 23:32:21.445250 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:21.445160 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f296a3b6-d7ef-477a-8221-b24528a9b8f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51110ea5-v7sdp\" (UID: \"f296a3b6-d7ef-477a-8221-b24528a9b8f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51110ea5-v7sdp" Nov 02 23:32:21.447139 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:21.447113 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f296a3b6-d7ef-477a-8221-b24528a9b8f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51110ea5-v7sdp\" (UID: \"f296a3b6-d7ef-477a-8221-b24528a9b8f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51110ea5-v7sdp" Nov 02 23:32:21.505370 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:21.505335 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7s7p\" (UniqueName: \"kubernetes.io/projected/f296a3b6-d7ef-477a-8221-b24528a9b8f4-kube-api-access-s7s7p\") pod \"maas-api-revoke-sub-51110ea5-v7sdp\" (UID: \"f296a3b6-d7ef-477a-8221-b24528a9b8f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51110ea5-v7sdp" Nov 02 23:32:21.602313 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:21.602218 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51110ea5-v7sdp" Nov 02 23:32:21.984347 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:21.984322 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51110ea5-v7sdp"] Nov 02 23:32:21.986596 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:32:21.986560 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf296a3b6_d7ef_477a_8221_b24528a9b8f4.slice/crio-8d1960e63f2ed7f43580d3b9b7c5837f9dc978522174e15f77a6ce8c10dd4bbe WatchSource:0}: Error finding container 8d1960e63f2ed7f43580d3b9b7c5837f9dc978522174e15f77a6ce8c10dd4bbe: Status 404 returned error can't find the container with id 8d1960e63f2ed7f43580d3b9b7c5837f9dc978522174e15f77a6ce8c10dd4bbe Nov 02 23:32:22.961256 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:22.961204 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51110ea5-v7sdp" event={"ID":"f296a3b6-d7ef-477a-8221-b24528a9b8f4","Type":"ContainerStarted","Data":"27be659ab2f130365fc4df079795066a8ce7265745208e24126e8a25bd251932"} Nov 02 23:32:22.961841 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:22.961289 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51110ea5-v7sdp" event={"ID":"f296a3b6-d7ef-477a-8221-b24528a9b8f4","Type":"ContainerStarted","Data":"8d1960e63f2ed7f43580d3b9b7c5837f9dc978522174e15f77a6ce8c10dd4bbe"} Nov 02 23:32:23.076275 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:23.076216 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51110ea5-v7sdp" podStartSLOduration=2.076200175 podStartE2EDuration="2.076200175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:23.07552368 +0000 UTC m=+2618.428557056" watchObservedRunningTime="2026-11-02 23:32:23.076200175 +0000 UTC m=+2618.429233569" Nov 02 23:32:23.968094 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:23.968055 5114 generic.go:356] "Generic (PLEG): container finished" podID="f296a3b6-d7ef-477a-8221-b24528a9b8f4" containerID="27be659ab2f130365fc4df079795066a8ce7265745208e24126e8a25bd251932" exitCode=0 Nov 02 23:32:23.968521 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:23.968144 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51110ea5-v7sdp" event={"ID":"f296a3b6-d7ef-477a-8221-b24528a9b8f4","Type":"ContainerDied","Data":"27be659ab2f130365fc4df079795066a8ce7265745208e24126e8a25bd251932"} Nov 02 23:32:25.125889 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:25.125856 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51110ea5-v7sdp" Nov 02 23:32:25.183261 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:25.183219 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f296a3b6-d7ef-477a-8221-b24528a9b8f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f296a3b6-d7ef-477a-8221-b24528a9b8f4-maas-api-openshift-ca\") pod \"f296a3b6-d7ef-477a-8221-b24528a9b8f4\" (UID: \"f296a3b6-d7ef-477a-8221-b24528a9b8f4\") " Nov 02 23:32:25.183445 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:25.183367 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f296a3b6-d7ef-477a-8221-b24528a9b8f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f296a3b6-d7ef-477a-8221-b24528a9b8f4-maas-api-xks-ca\") pod \"f296a3b6-d7ef-477a-8221-b24528a9b8f4\" (UID: \"f296a3b6-d7ef-477a-8221-b24528a9b8f4\") " Nov 02 23:32:25.183445 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:25.183441 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f296a3b6-d7ef-477a-8221-b24528a9b8f4-kube-api-access-s7s7p\" (UniqueName: \"kubernetes.io/projected/f296a3b6-d7ef-477a-8221-b24528a9b8f4-kube-api-access-s7s7p\") pod \"f296a3b6-d7ef-477a-8221-b24528a9b8f4\" (UID: \"f296a3b6-d7ef-477a-8221-b24528a9b8f4\") " Nov 02 23:32:25.183656 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:25.183634 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f296a3b6-d7ef-477a-8221-b24528a9b8f4-maas-api-openshift-ca" pod "f296a3b6-d7ef-477a-8221-b24528a9b8f4" (UID: "f296a3b6-d7ef-477a-8221-b24528a9b8f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:32:25.183874 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:25.183850 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f296a3b6-d7ef-477a-8221-b24528a9b8f4-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:25.185772 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:25.185744 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f296a3b6-d7ef-477a-8221-b24528a9b8f4-maas-api-xks-ca" pod "f296a3b6-d7ef-477a-8221-b24528a9b8f4" (UID: "f296a3b6-d7ef-477a-8221-b24528a9b8f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:32:25.185916 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:25.185829 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f296a3b6-d7ef-477a-8221-b24528a9b8f4-kube-api-access-s7s7p" pod "f296a3b6-d7ef-477a-8221-b24528a9b8f4" (UID: "f296a3b6-d7ef-477a-8221-b24528a9b8f4"). InnerVolumeSpecName "kube-api-access-s7s7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:32:25.284774 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:25.284738 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7s7p\" (UniqueName: \"kubernetes.io/projected/f296a3b6-d7ef-477a-8221-b24528a9b8f4-kube-api-access-s7s7p\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:25.284774 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:25.284775 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f296a3b6-d7ef-477a-8221-b24528a9b8f4-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:25.979900 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:25.979868 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51110ea5-v7sdp" Nov 02 23:32:25.980069 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:25.979866 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51110ea5-v7sdp" event={"ID":"f296a3b6-d7ef-477a-8221-b24528a9b8f4","Type":"ContainerDied","Data":"8d1960e63f2ed7f43580d3b9b7c5837f9dc978522174e15f77a6ce8c10dd4bbe"} Nov 02 23:32:25.980069 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:25.980009 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d1960e63f2ed7f43580d3b9b7c5837f9dc978522174e15f77a6ce8c10dd4bbe" Nov 02 23:32:26.946672 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:26.946629 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6"] Nov 02 23:32:26.947168 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:26.947051 5114 cpu_manager.go:414] "containerMap: removing container" podUID="f296a3b6-d7ef-477a-8221-b24528a9b8f4" containerName="revoke-keys" Nov 02 23:32:26.947168 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:26.947064 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f296a3b6-d7ef-477a-8221-b24528a9b8f4" containerName="revoke-keys" Nov 02 23:32:26.947168 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:26.947153 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="f296a3b6-d7ef-477a-8221-b24528a9b8f4" containerName="revoke-keys" Nov 02 23:32:26.950720 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:26.950694 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.003999 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.003965 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.003999 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.004004 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.004231 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.004027 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.004231 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.004101 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qk42\" (UniqueName: \"kubernetes.io/projected/5693f409-a9e5-487a-885a-c254f8ef63cd-kube-api-access-5qk42\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.004231 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.004156 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.004231 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.004200 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.004231 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.004226 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5693f409-a9e5-487a-885a-c254f8ef63cd-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.004638 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.004287 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.004638 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.004368 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.105444 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.105404 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5693f409-a9e5-487a-885a-c254f8ef63cd-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.105702 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.105452 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.105702 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.105586 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.105702 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.105653 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.105702 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.105679 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.105929 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.105707 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.105929 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.105736 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qk42\" (UniqueName: \"kubernetes.io/projected/5693f409-a9e5-487a-885a-c254f8ef63cd-kube-api-access-5qk42\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.105929 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.105875 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.106094 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.106049 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.106094 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.106067 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.106203 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.106177 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.106339 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.106317 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.106395 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.106308 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5693f409-a9e5-487a-885a-c254f8ef63cd-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.106704 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.106679 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.108315 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.108287 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.108454 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.108425 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.154402 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.154364 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6"] Nov 02 23:32:27.167582 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.167544 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.167582 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.167567 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qk42\" (UniqueName: \"kubernetes.io/projected/5693f409-a9e5-487a-885a-c254f8ef63cd-kube-api-access-5qk42\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.264892 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.264800 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:27.701940 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.701909 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6"] Nov 02 23:32:27.703985 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:32:27.703955 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5693f409_a9e5_487a_885a_c254f8ef63cd.slice/crio-529274da6bbeeb977a4e567ad30b054c36baf9f8947f384ab0c83f91d2b92b52 WatchSource:0}: Error finding container 529274da6bbeeb977a4e567ad30b054c36baf9f8947f384ab0c83f91d2b92b52: Status 404 returned error can't find the container with id 529274da6bbeeb977a4e567ad30b054c36baf9f8947f384ab0c83f91d2b92b52 Nov 02 23:32:27.706426 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.706391 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:32:27.706567 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.706502 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:32:27.706567 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.706551 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:32:27.993190 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.993091 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" event={"ID":"5693f409-a9e5-487a-885a-c254f8ef63cd","Type":"ContainerStarted","Data":"4d7874c4db39bdfdf031f1b47383a35c339ba4aa32f081cf9b9c9be8b46f11e3"} Nov 02 23:32:27.993190 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:27.993148 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" event={"ID":"5693f409-a9e5-487a-885a-c254f8ef63cd","Type":"ContainerStarted","Data":"529274da6bbeeb977a4e567ad30b054c36baf9f8947f384ab0c83f91d2b92b52"} Nov 02 23:32:28.115544 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:28.115456 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" podStartSLOduration=2.115431561 podStartE2EDuration="2.115431561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:28.113269658 +0000 UTC m=+2623.466303034" watchObservedRunningTime="2026-11-02 23:32:28.115431561 +0000 UTC m=+2623.468464937" Nov 02 23:32:28.265310 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:28.265225 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:28.266970 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:28.266943 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 23:32:28.267101 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:28.266995 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" podUID="5693f409-a9e5-487a-885a-c254f8ef63cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 23:32:29.265655 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:29.265615 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 23:32:29.266108 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:29.265681 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" podUID="5693f409-a9e5-487a-885a-c254f8ef63cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 23:32:30.265249 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:30.265211 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 23:32:30.265452 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:30.265283 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" podUID="5693f409-a9e5-487a-885a-c254f8ef63cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 23:32:31.265299 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:31.265260 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 23:32:31.265845 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:31.265319 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" podUID="5693f409-a9e5-487a-885a-c254f8ef63cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 23:32:32.265278 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:32.265239 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 23:32:32.265508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:32.265297 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" podUID="5693f409-a9e5-487a-885a-c254f8ef63cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 23:32:33.265818 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:33.265772 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 23:32:33.266338 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:33.265852 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" podUID="5693f409-a9e5-487a-885a-c254f8ef63cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 23:32:34.265233 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:34.265197 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 23:32:34.265426 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:34.265278 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" podUID="5693f409-a9e5-487a-885a-c254f8ef63cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 23:32:36.135513 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:36.131308 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" podUID="5e3275e5-f49f-4ca7-8d49-03178a77fee6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:32:36.262290 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:36.262224 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" podUID="72301e09-ed03-4bc9-8264-028de7930a48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 23:32:36.321656 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:36.321613 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:32:36.321964 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:36.321915 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" podUID="5693f409-a9e5-487a-885a-c254f8ef63cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:32:36.467459 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:36.467365 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:36.468691 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:36.468660 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:36.469625 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:36.469600 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:32:37.586157 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:32:37.586119 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:32:44.605873 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:44.605826 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-567e11bd-4rbsn"] Nov 02 23:32:44.609680 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:44.609645 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-567e11bd-4rbsn" Nov 02 23:32:44.664192 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:44.664158 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:32:44.713406 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:44.713376 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:32:44.715677 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:44.715645 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-567e11bd-4rbsn"] Nov 02 23:32:44.732254 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:44.732205 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htg4r\" (UniqueName: \"kubernetes.io/projected/1334b934-4cbb-405d-a48c-125af5e6b07d-kube-api-access-htg4r\") pod \"maas-api-revoke-sub-567e11bd-4rbsn\" (UID: \"1334b934-4cbb-405d-a48c-125af5e6b07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-567e11bd-4rbsn" Nov 02 23:32:44.732449 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:44.732351 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1334b934-4cbb-405d-a48c-125af5e6b07d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-567e11bd-4rbsn\" (UID: \"1334b934-4cbb-405d-a48c-125af5e6b07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-567e11bd-4rbsn" Nov 02 23:32:44.732449 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:44.732396 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1334b934-4cbb-405d-a48c-125af5e6b07d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-567e11bd-4rbsn\" (UID: \"1334b934-4cbb-405d-a48c-125af5e6b07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-567e11bd-4rbsn" Nov 02 23:32:44.833899 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:44.833863 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htg4r\" (UniqueName: \"kubernetes.io/projected/1334b934-4cbb-405d-a48c-125af5e6b07d-kube-api-access-htg4r\") pod \"maas-api-revoke-sub-567e11bd-4rbsn\" (UID: \"1334b934-4cbb-405d-a48c-125af5e6b07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-567e11bd-4rbsn" Nov 02 23:32:44.834088 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:44.833956 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1334b934-4cbb-405d-a48c-125af5e6b07d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-567e11bd-4rbsn\" (UID: \"1334b934-4cbb-405d-a48c-125af5e6b07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-567e11bd-4rbsn" Nov 02 23:32:44.834088 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:44.833989 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1334b934-4cbb-405d-a48c-125af5e6b07d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-567e11bd-4rbsn\" (UID: \"1334b934-4cbb-405d-a48c-125af5e6b07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-567e11bd-4rbsn" Nov 02 23:32:44.834740 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:44.834715 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1334b934-4cbb-405d-a48c-125af5e6b07d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-567e11bd-4rbsn\" (UID: \"1334b934-4cbb-405d-a48c-125af5e6b07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-567e11bd-4rbsn" Nov 02 23:32:44.885150 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:44.885067 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:32:44.894836 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:44.894747 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htg4r\" (UniqueName: \"kubernetes.io/projected/1334b934-4cbb-405d-a48c-125af5e6b07d-kube-api-access-htg4r\") pod \"maas-api-revoke-sub-567e11bd-4rbsn\" (UID: \"1334b934-4cbb-405d-a48c-125af5e6b07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-567e11bd-4rbsn" Nov 02 23:32:44.896586 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:44.896561 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1334b934-4cbb-405d-a48c-125af5e6b07d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-567e11bd-4rbsn\" (UID: \"1334b934-4cbb-405d-a48c-125af5e6b07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-567e11bd-4rbsn" Nov 02 23:32:44.975409 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:44.975373 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:32:44.984659 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:44.984623 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-567e11bd-4rbsn" Nov 02 23:32:45.360033 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:45.360004 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-567e11bd-4rbsn"] Nov 02 23:32:45.362421 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:32:45.362390 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1334b934_4cbb_405d_a48c_125af5e6b07d.slice/crio-6d70d181cf1224a80cdc9ba6f2f684f0ba8c407ec90ed62fa92cfd2ed13b0f06 WatchSource:0}: Error finding container 6d70d181cf1224a80cdc9ba6f2f684f0ba8c407ec90ed62fa92cfd2ed13b0f06: Status 404 returned error can't find the container with id 6d70d181cf1224a80cdc9ba6f2f684f0ba8c407ec90ed62fa92cfd2ed13b0f06 Nov 02 23:32:46.225007 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:46.224954 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-567e11bd-4rbsn" event={"ID":"1334b934-4cbb-405d-a48c-125af5e6b07d","Type":"ContainerStarted","Data":"66e3ddaea0a119da1a5a92a8725eb35771f664bcf97d30a4f45fa7554786cb60"} Nov 02 23:32:46.225568 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:46.225038 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-567e11bd-4rbsn" event={"ID":"1334b934-4cbb-405d-a48c-125af5e6b07d","Type":"ContainerStarted","Data":"6d70d181cf1224a80cdc9ba6f2f684f0ba8c407ec90ed62fa92cfd2ed13b0f06"} Nov 02 23:32:46.343503 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:46.343425 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-567e11bd-4rbsn" podStartSLOduration=2.343407359 podStartE2EDuration="2.343407359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:46.34137706 +0000 UTC m=+2641.694410434" watchObservedRunningTime="2026-11-02 23:32:46.343407359 +0000 UTC m=+2641.696440780" Nov 02 23:32:47.232156 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:47.232119 5114 generic.go:356] "Generic (PLEG): container finished" podID="1334b934-4cbb-405d-a48c-125af5e6b07d" containerID="66e3ddaea0a119da1a5a92a8725eb35771f664bcf97d30a4f45fa7554786cb60" exitCode=0 Nov 02 23:32:47.232608 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:47.232185 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-567e11bd-4rbsn" event={"ID":"1334b934-4cbb-405d-a48c-125af5e6b07d","Type":"ContainerDied","Data":"66e3ddaea0a119da1a5a92a8725eb35771f664bcf97d30a4f45fa7554786cb60"} Nov 02 23:32:48.381972 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:48.381934 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-567e11bd-4rbsn" Nov 02 23:32:48.476356 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:48.476312 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1334b934-4cbb-405d-a48c-125af5e6b07d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1334b934-4cbb-405d-a48c-125af5e6b07d-maas-api-openshift-ca\") pod \"1334b934-4cbb-405d-a48c-125af5e6b07d\" (UID: \"1334b934-4cbb-405d-a48c-125af5e6b07d\") " Nov 02 23:32:48.476605 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:48.476517 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1334b934-4cbb-405d-a48c-125af5e6b07d-kube-api-access-htg4r\" (UniqueName: \"kubernetes.io/projected/1334b934-4cbb-405d-a48c-125af5e6b07d-kube-api-access-htg4r\") pod \"1334b934-4cbb-405d-a48c-125af5e6b07d\" (UID: \"1334b934-4cbb-405d-a48c-125af5e6b07d\") " Nov 02 23:32:48.476605 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:48.476584 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1334b934-4cbb-405d-a48c-125af5e6b07d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1334b934-4cbb-405d-a48c-125af5e6b07d-maas-api-xks-ca\") pod \"1334b934-4cbb-405d-a48c-125af5e6b07d\" (UID: \"1334b934-4cbb-405d-a48c-125af5e6b07d\") " Nov 02 23:32:48.476828 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:48.476800 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1334b934-4cbb-405d-a48c-125af5e6b07d-maas-api-openshift-ca" pod "1334b934-4cbb-405d-a48c-125af5e6b07d" (UID: "1334b934-4cbb-405d-a48c-125af5e6b07d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:32:48.477057 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:48.477038 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1334b934-4cbb-405d-a48c-125af5e6b07d-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:48.478965 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:48.478935 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1334b934-4cbb-405d-a48c-125af5e6b07d-maas-api-xks-ca" pod "1334b934-4cbb-405d-a48c-125af5e6b07d" (UID: "1334b934-4cbb-405d-a48c-125af5e6b07d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:32:48.479060 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:48.478988 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1334b934-4cbb-405d-a48c-125af5e6b07d-kube-api-access-htg4r" pod "1334b934-4cbb-405d-a48c-125af5e6b07d" (UID: "1334b934-4cbb-405d-a48c-125af5e6b07d"). InnerVolumeSpecName "kube-api-access-htg4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:32:48.578162 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:48.578126 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htg4r\" (UniqueName: \"kubernetes.io/projected/1334b934-4cbb-405d-a48c-125af5e6b07d-kube-api-access-htg4r\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:48.578162 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:48.578162 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1334b934-4cbb-405d-a48c-125af5e6b07d-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:49.248326 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:49.248293 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-567e11bd-4rbsn" Nov 02 23:32:49.248535 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:49.248320 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-567e11bd-4rbsn" event={"ID":"1334b934-4cbb-405d-a48c-125af5e6b07d","Type":"ContainerDied","Data":"6d70d181cf1224a80cdc9ba6f2f684f0ba8c407ec90ed62fa92cfd2ed13b0f06"} Nov 02 23:32:49.248535 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:49.248372 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d70d181cf1224a80cdc9ba6f2f684f0ba8c407ec90ed62fa92cfd2ed13b0f06" Nov 02 23:32:57.185958 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.185917 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj"] Nov 02 23:32:57.186651 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.186630 5114 cpu_manager.go:414] "containerMap: removing container" podUID="1334b934-4cbb-405d-a48c-125af5e6b07d" containerName="revoke-keys" Nov 02 23:32:57.186756 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.186655 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1334b934-4cbb-405d-a48c-125af5e6b07d" containerName="revoke-keys" Nov 02 23:32:57.186826 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.186813 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="1334b934-4cbb-405d-a48c-125af5e6b07d" containerName="revoke-keys" Nov 02 23:32:57.190798 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.190772 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.284067 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.284000 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.284287 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.284086 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9840bb1-3a88-4d36-8077-1facecc81158-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.284287 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.284185 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.284429 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.284293 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.284429 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.284323 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.284429 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.284364 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9840bb1-3a88-4d36-8077-1facecc81158-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.284429 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.284396 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9840bb1-3a88-4d36-8077-1facecc81158-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.284710 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.284675 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.284862 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.284714 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8v8q\" (UniqueName: \"kubernetes.io/projected/a9840bb1-3a88-4d36-8077-1facecc81158-kube-api-access-m8v8q\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.295153 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.295108 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj"] Nov 02 23:32:57.385923 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.385885 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.385923 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.385927 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.386234 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.385978 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9840bb1-3a88-4d36-8077-1facecc81158-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.386234 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.386018 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9840bb1-3a88-4d36-8077-1facecc81158-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.386457 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.386433 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.386721 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.386686 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.386851 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.386733 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8v8q\" (UniqueName: \"kubernetes.io/projected/a9840bb1-3a88-4d36-8077-1facecc81158-kube-api-access-m8v8q\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.386926 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.386873 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.387149 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.386961 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9840bb1-3a88-4d36-8077-1facecc81158-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.387149 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.386969 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.387262 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.387174 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.387262 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.387247 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.387437 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.386873 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9840bb1-3a88-4d36-8077-1facecc81158-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.387571 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.387541 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.389092 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.389068 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9840bb1-3a88-4d36-8077-1facecc81158-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.389875 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.389855 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.449728 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.449649 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9840bb1-3a88-4d36-8077-1facecc81158-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.449888 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.449860 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8v8q\" (UniqueName: \"kubernetes.io/projected/a9840bb1-3a88-4d36-8077-1facecc81158-kube-api-access-m8v8q\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.505534 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.505495 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:57.953818 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.950072 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj"] Nov 02 23:32:57.957187 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.957139 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:32:57.957325 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.957246 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:32:57.957325 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:57.957291 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:32:58.322911 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:58.320119 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" event={"ID":"a9840bb1-3a88-4d36-8077-1facecc81158","Type":"ContainerStarted","Data":"69b2049e9a94810b8fb40bf3935951ae1d1360195580d06fc89e68ff40350050"} Nov 02 23:32:58.323757 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:58.323724 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" event={"ID":"a9840bb1-3a88-4d36-8077-1facecc81158","Type":"ContainerStarted","Data":"53cb9777d5822f9b2a4f787b07fca63dad529f72005dfe8ce0250a22eb265341"} Nov 02 23:32:58.533512 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:58.515088 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:32:59.514510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:59.513623 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 23:32:59.514510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:32:59.513691 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" podUID="a9840bb1-3a88-4d36-8077-1facecc81158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 23:33:00.522217 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:00.509920 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 23:33:00.522217 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:00.510005 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" podUID="a9840bb1-3a88-4d36-8077-1facecc81158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 23:33:01.513510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:01.507954 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 23:33:01.513510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:01.508033 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" podUID="a9840bb1-3a88-4d36-8077-1facecc81158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 23:33:02.253444 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:02.253378 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" podStartSLOduration=5.25336014 podStartE2EDuration="5.25336014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:58.515024131 +0000 UTC m=+2653.868057541" watchObservedRunningTime="2026-11-02 23:33:02.25336014 +0000 UTC m=+2657.606393514" Nov 02 23:33:02.254869 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:02.254845 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87d506e2-mgpn8"] Nov 02 23:33:02.258842 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:02.258824 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d506e2-mgpn8" Nov 02 23:33:02.305818 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:02.305542 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87d506e2-mgpn8\" (UID: \"80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d506e2-mgpn8" Nov 02 23:33:02.305818 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:02.305643 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87d506e2-mgpn8\" (UID: \"80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d506e2-mgpn8" Nov 02 23:33:02.305818 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:02.305726 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft5hw\" (UniqueName: \"kubernetes.io/projected/80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8-kube-api-access-ft5hw\") pod \"maas-api-revoke-sub-87d506e2-mgpn8\" (UID: \"80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d506e2-mgpn8" Nov 02 23:33:02.322505 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:02.320215 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:33:02.370142 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:02.370109 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:33:02.372541 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:02.372513 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87d506e2-mgpn8"] Nov 02 23:33:02.430808 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:02.430750 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft5hw\" (UniqueName: \"kubernetes.io/projected/80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8-kube-api-access-ft5hw\") pod \"maas-api-revoke-sub-87d506e2-mgpn8\" (UID: \"80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d506e2-mgpn8" Nov 02 23:33:02.431021 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:02.430935 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87d506e2-mgpn8\" (UID: \"80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d506e2-mgpn8" Nov 02 23:33:02.431091 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:02.431022 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87d506e2-mgpn8\" (UID: \"80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d506e2-mgpn8" Nov 02 23:33:02.431914 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:02.431884 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87d506e2-mgpn8\" (UID: \"80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d506e2-mgpn8" Nov 02 23:33:02.433884 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:02.433858 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87d506e2-mgpn8\" (UID: \"80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d506e2-mgpn8" Nov 02 23:33:02.491812 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:02.491777 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft5hw\" (UniqueName: \"kubernetes.io/projected/80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8-kube-api-access-ft5hw\") pod \"maas-api-revoke-sub-87d506e2-mgpn8\" (UID: \"80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d506e2-mgpn8" Nov 02 23:33:02.528027 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:02.527929 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 23:33:02.528311 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:02.528280 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" podUID="a9840bb1-3a88-4d36-8077-1facecc81158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 23:33:02.574192 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:02.574146 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d506e2-mgpn8" Nov 02 23:33:02.974541 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:02.974511 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87d506e2-mgpn8"] Nov 02 23:33:02.976696 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:33:02.976655 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d8cf2e_9b7d_42b0_a7f8_3e216b885ce8.slice/crio-4a2da787825038640b959a6091c7ff37886e0d80f39112e071e2225985016a78 WatchSource:0}: Error finding container 4a2da787825038640b959a6091c7ff37886e0d80f39112e071e2225985016a78: Status 404 returned error can't find the container with id 4a2da787825038640b959a6091c7ff37886e0d80f39112e071e2225985016a78 Nov 02 23:33:03.375890 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:03.375857 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d506e2-mgpn8" event={"ID":"80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8","Type":"ContainerStarted","Data":"4a2da787825038640b959a6091c7ff37886e0d80f39112e071e2225985016a78"} Nov 02 23:33:03.527567 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:03.527382 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 23:33:03.527567 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:03.527452 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" podUID="a9840bb1-3a88-4d36-8077-1facecc81158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 23:33:04.382889 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:04.382860 5114 generic.go:356] "Generic (PLEG): container finished" podID="80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8" containerID="9b3431e05c7373352f50390c7af249aed18557608b770a7b6c35c638f7a6e101" exitCode=0 Nov 02 23:33:04.383402 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:04.382988 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d506e2-mgpn8" event={"ID":"80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8","Type":"ContainerDied","Data":"9b3431e05c7373352f50390c7af249aed18557608b770a7b6c35c638f7a6e101"} Nov 02 23:33:04.524604 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:04.512906 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 23:33:04.524604 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:04.512978 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" podUID="a9840bb1-3a88-4d36-8077-1facecc81158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 23:33:05.506376 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:05.506334 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 23:33:05.506902 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:05.506408 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" podUID="a9840bb1-3a88-4d36-8077-1facecc81158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 23:33:05.543119 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:05.543094 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d506e2-mgpn8" Nov 02 23:33:05.670733 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:05.670626 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8-maas-api-xks-ca\") pod \"80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8\" (UID: \"80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8\") " Nov 02 23:33:05.670733 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:05.670676 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8-kube-api-access-ft5hw\" (UniqueName: \"kubernetes.io/projected/80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8-kube-api-access-ft5hw\") pod \"80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8\" (UID: \"80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8\") " Nov 02 23:33:05.670996 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:05.670803 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8-maas-api-openshift-ca\") pod \"80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8\" (UID: \"80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8\") " Nov 02 23:33:05.671826 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:05.671794 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8-maas-api-openshift-ca" pod "80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8" (UID: "80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:33:05.674140 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:05.674100 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8-maas-api-xks-ca" pod "80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8" (UID: "80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:33:05.674279 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:05.674144 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8-kube-api-access-ft5hw" pod "80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8" (UID: "80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8"). InnerVolumeSpecName "kube-api-access-ft5hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:33:05.772390 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:05.772346 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:33:05.772390 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:05.772377 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft5hw\" (UniqueName: \"kubernetes.io/projected/80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8-kube-api-access-ft5hw\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:33:05.772390 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:05.772388 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:33:06.395848 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:06.395803 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d506e2-mgpn8" event={"ID":"80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8","Type":"ContainerDied","Data":"4a2da787825038640b959a6091c7ff37886e0d80f39112e071e2225985016a78"} Nov 02 23:33:06.396026 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:06.395870 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a2da787825038640b959a6091c7ff37886e0d80f39112e071e2225985016a78" Nov 02 23:33:06.396026 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:06.395871 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d506e2-mgpn8" Nov 02 23:33:06.511178 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:06.511142 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:33:06.511700 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:06.511528 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:33:06.512408 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:06.512387 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:33:07.583296 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:33:07.583247 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:33:22.820420 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:22.820379 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv"] Nov 02 23:33:22.820859 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:22.820842 5114 cpu_manager.go:414] "containerMap: removing container" podUID="80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8" containerName="revoke-keys" Nov 02 23:33:22.820907 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:22.820856 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8" containerName="revoke-keys" Nov 02 23:33:22.820971 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:22.820951 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8" containerName="revoke-keys" Nov 02 23:33:22.824208 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:22.824192 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" Nov 02 23:33:22.876384 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:22.876346 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-94e3ae-serving-cert\"" Nov 02 23:33:22.921294 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:22.921258 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv\" (UID: \"8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" Nov 02 23:33:22.921538 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:22.921335 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv\" (UID: \"8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" Nov 02 23:33:22.922072 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:22.922044 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkgs7\" (UniqueName: \"kubernetes.io/projected/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-kube-api-access-rkgs7\") pod \"maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv\" (UID: \"8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" Nov 02 23:33:22.922210 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:22.922090 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-metrics-certs\") pod \"maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv\" (UID: \"8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" Nov 02 23:33:22.928680 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:22.928640 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv"] Nov 02 23:33:23.023126 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:23.023080 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv\" (UID: \"8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" Nov 02 23:33:23.023301 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:23.023155 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv\" (UID: \"8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" Nov 02 23:33:23.023301 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:23.023265 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkgs7\" (UniqueName: \"kubernetes.io/projected/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-kube-api-access-rkgs7\") pod \"maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv\" (UID: \"8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" Nov 02 23:33:23.023301 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:23.023286 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-metrics-certs\") pod \"maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv\" (UID: \"8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" Nov 02 23:33:23.023451 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:33:23.023299 5114 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-serving-cert: secret "maas-api-e2e-praxis-a-94e3ae-serving-cert" not found Nov 02 23:33:23.023451 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:33:23.023368 5114 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-maas-api-tls podName:8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5 nodeName:}" failed. No retries permitted until 2026-11-02 23:33:23.523348869 +0000 UTC m=+2678.876382221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-maas-api-tls") pod "maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" (UID: "8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5") : secret "maas-api-e2e-praxis-a-94e3ae-serving-cert" not found Nov 02 23:33:23.023797 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:23.023777 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv\" (UID: \"8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" Nov 02 23:33:23.025929 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:23.025907 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-metrics-certs\") pod \"maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv\" (UID: \"8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" Nov 02 23:33:23.084635 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:23.084558 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkgs7\" (UniqueName: \"kubernetes.io/projected/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-kube-api-access-rkgs7\") pod \"maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv\" (UID: \"8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" Nov 02 23:33:23.531259 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:23.531152 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv\" (UID: \"8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" Nov 02 23:33:23.533971 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:23.533940 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv\" (UID: \"8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" Nov 02 23:33:23.738015 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:23.737984 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" Nov 02 23:33:24.120203 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:24.120167 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv"] Nov 02 23:33:24.121300 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:33:24.121272 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8892cb6b_3a9d_4b85_9d48_d4a7f89b78a5.slice/crio-911fdaad888322d56e136737e8623b01a1308782700ffae4f02cfc2777011fce WatchSource:0}: Error finding container 911fdaad888322d56e136737e8623b01a1308782700ffae4f02cfc2777011fce: Status 404 returned error can't find the container with id 911fdaad888322d56e136737e8623b01a1308782700ffae4f02cfc2777011fce Nov 02 23:33:24.557714 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:24.557678 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" event={"ID":"8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5","Type":"ContainerStarted","Data":"911fdaad888322d56e136737e8623b01a1308782700ffae4f02cfc2777011fce"} Nov 02 23:33:25.564976 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:25.564926 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" event={"ID":"8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5","Type":"ContainerStarted","Data":"e30dcd68a7c825de8809dad376c20016408b4ad1c5f9116e26f04b5f5717d75e"} Nov 02 23:33:25.565530 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:25.565004 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" Nov 02 23:33:25.683987 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:25.683934 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" podStartSLOduration=2.793639527 podStartE2EDuration="3.683919367s" totalImagesPullingTime="890.27984ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:33:25.681930889 +0000 UTC m=+2681.034964265" watchObservedRunningTime="2026-11-02 23:33:25.683919367 +0000 UTC m=+2681.036952741" Nov 02 23:33:31.580109 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:31.580080 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" Nov 02 23:33:37.583732 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:33:37.583693 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:33:54.017056 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:54.016990 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h"] Nov 02 23:33:54.020992 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:54.020953 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h" Nov 02 23:33:54.093160 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:54.093110 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f7747ad-4a88-41c8-95d2-bfac505c05ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h\" (UID: \"8f7747ad-4a88-41c8-95d2-bfac505c05ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h" Nov 02 23:33:54.093338 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:54.093250 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwdcd\" (UniqueName: \"kubernetes.io/projected/8f7747ad-4a88-41c8-95d2-bfac505c05ca-kube-api-access-qwdcd\") pod \"payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h\" (UID: \"8f7747ad-4a88-41c8-95d2-bfac505c05ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h" Nov 02 23:33:54.124774 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:54.124729 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2279ad\"" Nov 02 23:33:54.125012 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:54.124796 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2279ad-dockercfg-8fpjv\"" Nov 02 23:33:54.127498 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:54.127450 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h"] Nov 02 23:33:54.194539 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:54.194501 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwdcd\" (UniqueName: \"kubernetes.io/projected/8f7747ad-4a88-41c8-95d2-bfac505c05ca-kube-api-access-qwdcd\") pod \"payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h\" (UID: \"8f7747ad-4a88-41c8-95d2-bfac505c05ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h" Nov 02 23:33:54.194740 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:54.194663 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f7747ad-4a88-41c8-95d2-bfac505c05ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h\" (UID: \"8f7747ad-4a88-41c8-95d2-bfac505c05ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h" Nov 02 23:33:54.195381 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:54.195352 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f7747ad-4a88-41c8-95d2-bfac505c05ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h\" (UID: \"8f7747ad-4a88-41c8-95d2-bfac505c05ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h" Nov 02 23:33:54.256690 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:54.256652 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwdcd\" (UniqueName: \"kubernetes.io/projected/8f7747ad-4a88-41c8-95d2-bfac505c05ca-kube-api-access-qwdcd\") pod \"payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h\" (UID: \"8f7747ad-4a88-41c8-95d2-bfac505c05ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h" Nov 02 23:33:54.337999 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:54.337901 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h" Nov 02 23:33:54.715312 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:54.715279 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h"] Nov 02 23:33:54.717256 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:33:54.717224 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7747ad_4a88_41c8_95d2_bfac505c05ca.slice/crio-02b5b41a322525b9928884b14e34bcfaa1c631359e17fd8c00470531fad91143 WatchSource:0}: Error finding container 02b5b41a322525b9928884b14e34bcfaa1c631359e17fd8c00470531fad91143: Status 404 returned error can't find the container with id 02b5b41a322525b9928884b14e34bcfaa1c631359e17fd8c00470531fad91143 Nov 02 23:33:54.741427 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:54.741391 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h" event={"ID":"8f7747ad-4a88-41c8-95d2-bfac505c05ca","Type":"ContainerStarted","Data":"02b5b41a322525b9928884b14e34bcfaa1c631359e17fd8c00470531fad91143"} Nov 02 23:33:54.819831 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:54.819792 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2"] Nov 02 23:33:54.823575 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:54.823550 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2" Nov 02 23:33:54.883388 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:54.883350 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2"] Nov 02 23:33:54.905393 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:54.905355 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6b0052-2d10-47d5-ae06-95d5fc77de1f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2\" (UID: \"1a6b0052-2d10-47d5-ae06-95d5fc77de1f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2" Nov 02 23:33:54.905655 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:54.905420 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqlnf\" (UniqueName: \"kubernetes.io/projected/1a6b0052-2d10-47d5-ae06-95d5fc77de1f-kube-api-access-gqlnf\") pod \"payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2\" (UID: \"1a6b0052-2d10-47d5-ae06-95d5fc77de1f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2" Nov 02 23:33:55.006387 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:55.006286 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6b0052-2d10-47d5-ae06-95d5fc77de1f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2\" (UID: \"1a6b0052-2d10-47d5-ae06-95d5fc77de1f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2" Nov 02 23:33:55.006580 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:55.006388 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqlnf\" (UniqueName: \"kubernetes.io/projected/1a6b0052-2d10-47d5-ae06-95d5fc77de1f-kube-api-access-gqlnf\") pod \"payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2\" (UID: \"1a6b0052-2d10-47d5-ae06-95d5fc77de1f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2" Nov 02 23:33:55.007259 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:55.007239 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6b0052-2d10-47d5-ae06-95d5fc77de1f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2\" (UID: \"1a6b0052-2d10-47d5-ae06-95d5fc77de1f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2" Nov 02 23:33:55.068042 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:55.068008 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqlnf\" (UniqueName: \"kubernetes.io/projected/1a6b0052-2d10-47d5-ae06-95d5fc77de1f-kube-api-access-gqlnf\") pod \"payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2\" (UID: \"1a6b0052-2d10-47d5-ae06-95d5fc77de1f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2" Nov 02 23:33:55.137786 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:55.137740 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2" Nov 02 23:33:55.515523 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:55.515475 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2"] Nov 02 23:33:55.517353 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:33:55.517318 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6b0052_2d10_47d5_ae06_95d5fc77de1f.slice/crio-7965001ac88378f2ba504aea60ab8e9d0150e89440fc19984d22b9b609bcdbee WatchSource:0}: Error finding container 7965001ac88378f2ba504aea60ab8e9d0150e89440fc19984d22b9b609bcdbee: Status 404 returned error can't find the container with id 7965001ac88378f2ba504aea60ab8e9d0150e89440fc19984d22b9b609bcdbee Nov 02 23:33:55.749673 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:55.749621 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h" event={"ID":"8f7747ad-4a88-41c8-95d2-bfac505c05ca","Type":"ContainerStarted","Data":"0f331948c34489cd90769c6884515c5540dd27d8207c30ff068ac71b604aa9b8"} Nov 02 23:33:55.749878 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:55.749697 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h" Nov 02 23:33:55.751244 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:55.751208 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2" event={"ID":"1a6b0052-2d10-47d5-ae06-95d5fc77de1f","Type":"ContainerStarted","Data":"2e2f8a9fa2ca950f3f7096189ad42dec62a73a7b979d87b2d5afcea10e7e5dfe"} Nov 02 23:33:55.751408 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:55.751281 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2" Nov 02 23:33:55.751408 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:55.751313 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2" event={"ID":"1a6b0052-2d10-47d5-ae06-95d5fc77de1f","Type":"ContainerStarted","Data":"7965001ac88378f2ba504aea60ab8e9d0150e89440fc19984d22b9b609bcdbee"} Nov 02 23:33:55.866595 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:55.866539 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h" podStartSLOduration=2.866525175 podStartE2EDuration="2.866525175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:33:55.864513456 +0000 UTC m=+2711.217546832" watchObservedRunningTime="2026-11-02 23:33:55.866525175 +0000 UTC m=+2711.219558550" Nov 02 23:33:55.982052 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:33:55.981991 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2" podStartSLOduration=1.981974358 podStartE2EDuration="1.981974358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:33:55.980760372 +0000 UTC m=+2711.333793773" watchObservedRunningTime="2026-11-02 23:33:55.981974358 +0000 UTC m=+2711.335007726" Nov 02 23:34:06.758870 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:06.758835 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2" Nov 02 23:34:06.759323 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:06.758918 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h" Nov 02 23:34:07.584263 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:34:07.584225 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:34:37.585327 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:34:37.585287 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:34:57.194603 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.194557 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh"] Nov 02 23:34:57.198330 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.198309 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.331508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.328405 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-915ccb-openshift-default-dockercfg-mm8vh\"" Nov 02 23:34:57.337167 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.332722 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh"] Nov 02 23:34:57.354410 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.354379 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istiod-ca-cert\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.354752 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.354733 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68ndw\" (UniqueName: \"kubernetes.io/projected/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-kube-api-access-68ndw\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.354919 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.354894 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-credential-socket\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.355032 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.355020 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-data\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.355159 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.355148 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-envoy\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.355253 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.355242 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-token\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.355349 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.355337 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-workload-socket\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.355452 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.355441 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-workload-certs\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.355591 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.355578 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-podinfo\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.456578 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.456461 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-data\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.456578 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.456551 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-envoy\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.456578 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.456567 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-token\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.456886 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.456590 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-workload-socket\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.456886 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.456619 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-workload-certs\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.456886 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.456650 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-podinfo\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.456886 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.456728 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istiod-ca-cert\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.456886 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.456810 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68ndw\" (UniqueName: \"kubernetes.io/projected/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-kube-api-access-68ndw\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.457192 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.456887 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-credential-socket\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.457192 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.456950 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-data\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.457192 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.457056 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-workload-certs\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.457339 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.457297 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-workload-socket\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.457576 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.457555 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-credential-socket\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.457796 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.457771 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istiod-ca-cert\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.459835 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.459807 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-podinfo\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.460089 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.460071 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-envoy\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.518100 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.518057 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-token\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.518269 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.518067 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68ndw\" (UniqueName: \"kubernetes.io/projected/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-kube-api-access-68ndw\") pod \"e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:57.812291 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:57.812246 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:58.275073 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:58.275039 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh"] Nov 02 23:34:58.277263 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:34:58.277226 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55da13ab_fe6e_4a9c_a7a0_ecc82bfb4046.slice/crio-c2ce116c4e1aa3f237d18c88b35e90bd1e8f0f7a120fcf792aef5611fafae206 WatchSource:0}: Error finding container c2ce116c4e1aa3f237d18c88b35e90bd1e8f0f7a120fcf792aef5611fafae206: Status 404 returned error can't find the container with id c2ce116c4e1aa3f237d18c88b35e90bd1e8f0f7a120fcf792aef5611fafae206 Nov 02 23:34:58.280126 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:58.280081 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:34:58.280235 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:58.280211 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:34:58.280303 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:58.280256 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:34:59.133642 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:59.133593 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" event={"ID":"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046","Type":"ContainerStarted","Data":"3800a86b7c6190d0fb87f7bef48260406279358468ecad3354f8cafaaf7286bb"} Nov 02 23:34:59.133877 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:59.133658 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" event={"ID":"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046","Type":"ContainerStarted","Data":"c2ce116c4e1aa3f237d18c88b35e90bd1e8f0f7a120fcf792aef5611fafae206"} Nov 02 23:34:59.253943 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:59.253881 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" podStartSLOduration=2.253865067 podStartE2EDuration="2.253865067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:34:59.252268045 +0000 UTC m=+2774.605301606" watchObservedRunningTime="2026-11-02 23:34:59.253865067 +0000 UTC m=+2774.606898493" Nov 02 23:34:59.813142 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:59.813098 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:34:59.818947 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:34:59.818920 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:35:00.139651 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:00.139564 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:35:00.140584 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:00.140561 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:35:07.583747 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:35:07.583702 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:35:23.665435 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:23.665388 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s"] Nov 02 23:35:23.666155 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:23.665683 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s" podUID="8bd9d853-a244-4206-b198-24652ae68088" containerName="payload-processing" containerID="cri-o://b00a6960d04d8049ac760168b4f1db8d0ab9eb2743932cf342f9a6e5958fd541" gracePeriod=30 Nov 02 23:35:23.957748 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:23.957722 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s" Nov 02 23:35:23.979662 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:23.979629 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bd9d853-a244-4206-b198-24652ae68088-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd9d853-a244-4206-b198-24652ae68088-plugins-config-volume\") pod \"8bd9d853-a244-4206-b198-24652ae68088\" (UID: \"8bd9d853-a244-4206-b198-24652ae68088\") " Nov 02 23:35:23.979800 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:23.979717 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bd9d853-a244-4206-b198-24652ae68088-kube-api-access-4bczh\" (UniqueName: \"kubernetes.io/projected/8bd9d853-a244-4206-b198-24652ae68088-kube-api-access-4bczh\") pod \"8bd9d853-a244-4206-b198-24652ae68088\" (UID: \"8bd9d853-a244-4206-b198-24652ae68088\") " Nov 02 23:35:23.980019 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:23.979997 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd9d853-a244-4206-b198-24652ae68088-plugins-config-volume" pod "8bd9d853-a244-4206-b198-24652ae68088" (UID: "8bd9d853-a244-4206-b198-24652ae68088"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:35:23.982057 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:23.982037 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd9d853-a244-4206-b198-24652ae68088-kube-api-access-4bczh" pod "8bd9d853-a244-4206-b198-24652ae68088" (UID: "8bd9d853-a244-4206-b198-24652ae68088"). InnerVolumeSpecName "kube-api-access-4bczh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:24.080755 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:24.080721 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bczh\" (UniqueName: \"kubernetes.io/projected/8bd9d853-a244-4206-b198-24652ae68088-kube-api-access-4bczh\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:24.080755 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:24.080751 5114 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd9d853-a244-4206-b198-24652ae68088-plugins-config-volume\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:24.281748 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:24.281650 5114 generic.go:356] "Generic (PLEG): container finished" podID="8bd9d853-a244-4206-b198-24652ae68088" containerID="b00a6960d04d8049ac760168b4f1db8d0ab9eb2743932cf342f9a6e5958fd541" exitCode=0 Nov 02 23:35:24.281748 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:24.281720 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s" Nov 02 23:35:24.281963 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:24.281750 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s" event={"ID":"8bd9d853-a244-4206-b198-24652ae68088","Type":"ContainerDied","Data":"b00a6960d04d8049ac760168b4f1db8d0ab9eb2743932cf342f9a6e5958fd541"} Nov 02 23:35:24.281963 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:24.281816 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s" event={"ID":"8bd9d853-a244-4206-b198-24652ae68088","Type":"ContainerDied","Data":"a2836beb05281ff1e29809bf4db56274814d63ac32ec87b7f84b5757bf22a29d"} Nov 02 23:35:24.281963 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:24.281848 5114 scope.go:122] "RemoveContainer" containerID="b00a6960d04d8049ac760168b4f1db8d0ab9eb2743932cf342f9a6e5958fd541" Nov 02 23:35:24.292811 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:24.292789 5114 scope.go:122] "RemoveContainer" containerID="b00a6960d04d8049ac760168b4f1db8d0ab9eb2743932cf342f9a6e5958fd541" Nov 02 23:35:24.293161 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:35:24.293139 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b00a6960d04d8049ac760168b4f1db8d0ab9eb2743932cf342f9a6e5958fd541\": container with ID starting with b00a6960d04d8049ac760168b4f1db8d0ab9eb2743932cf342f9a6e5958fd541 not found: ID does not exist" containerID="b00a6960d04d8049ac760168b4f1db8d0ab9eb2743932cf342f9a6e5958fd541" Nov 02 23:35:24.293210 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:24.293172 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00a6960d04d8049ac760168b4f1db8d0ab9eb2743932cf342f9a6e5958fd541"} err="failed to get container status \"b00a6960d04d8049ac760168b4f1db8d0ab9eb2743932cf342f9a6e5958fd541\": rpc error: code = NotFound desc = could not find container \"b00a6960d04d8049ac760168b4f1db8d0ab9eb2743932cf342f9a6e5958fd541\": container with ID starting with b00a6960d04d8049ac760168b4f1db8d0ab9eb2743932cf342f9a6e5958fd541 not found: ID does not exist" Nov 02 23:35:24.445972 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:24.445925 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s"] Nov 02 23:35:24.449326 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:24.449297 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-818fe8-5d869d4ddb-qdq2s"] Nov 02 23:35:24.795325 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:24.795284 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bd9d853-a244-4206-b198-24652ae68088" path="/var/lib/kubelet/pods/8bd9d853-a244-4206-b198-24652ae68088/volumes" Nov 02 23:35:24.959349 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:24.959309 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl"] Nov 02 23:35:24.959590 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:24.959555 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" podUID="afc01b1c-b566-4d06-bd25-c99d8b60ac3c" containerName="istio-proxy" containerID="cri-o://edc8c2aafdae525b4707f98ffb42c81facfe701d759b799e10d97058529742d0" gracePeriod=30 Nov 02 23:35:26.235391 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.235350 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p"] Nov 02 23:35:26.235876 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.235831 5114 cpu_manager.go:414] "containerMap: removing container" podUID="8bd9d853-a244-4206-b198-24652ae68088" containerName="payload-processing" Nov 02 23:35:26.235876 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.235844 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bd9d853-a244-4206-b198-24652ae68088" containerName="payload-processing" Nov 02 23:35:26.235964 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.235933 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bd9d853-a244-4206-b198-24652ae68088" containerName="payload-processing" Nov 02 23:35:26.240583 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.240557 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:35:26.293048 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.293019 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 23:35:26.293048 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.293023 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 23:35:26.293289 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.293054 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rdqkr\"" Nov 02 23:35:26.293289 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.293026 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 23:35:26.303329 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.303288 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cbx4p\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:35:26.303329 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.303329 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cbx4p\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:35:26.303607 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.303353 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f195682-c9a3-4e14-994c-5f795c95f41a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cbx4p\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:35:26.303607 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.303428 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2vnp\" (UniqueName: \"kubernetes.io/projected/1f195682-c9a3-4e14-994c-5f795c95f41a-kube-api-access-n2vnp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cbx4p\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:35:26.303607 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.303543 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1f195682-c9a3-4e14-994c-5f795c95f41a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cbx4p\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:35:26.303607 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.303561 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cbx4p\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:35:26.303759 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.303648 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cbx4p\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:35:26.346379 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.346344 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p"] Nov 02 23:35:26.404728 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.404689 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cbx4p\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:35:26.404940 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.404790 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cbx4p\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:35:26.404940 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.404812 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cbx4p\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:35:26.404940 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.404827 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f195682-c9a3-4e14-994c-5f795c95f41a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cbx4p\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:35:26.404940 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.404856 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2vnp\" (UniqueName: \"kubernetes.io/projected/1f195682-c9a3-4e14-994c-5f795c95f41a-kube-api-access-n2vnp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cbx4p\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:35:26.404940 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.404934 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1f195682-c9a3-4e14-994c-5f795c95f41a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cbx4p\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:35:26.405216 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.404952 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cbx4p\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:35:26.406005 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.405971 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cbx4p\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:35:26.407544 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.407477 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1f195682-c9a3-4e14-994c-5f795c95f41a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cbx4p\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:35:26.407664 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.407552 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f195682-c9a3-4e14-994c-5f795c95f41a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cbx4p\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:35:26.407664 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.407602 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cbx4p\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:35:26.407741 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.407695 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cbx4p\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:35:26.465661 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.465618 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cbx4p\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:35:26.465947 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.465919 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2vnp\" (UniqueName: \"kubernetes.io/projected/1f195682-c9a3-4e14-994c-5f795c95f41a-kube-api-access-n2vnp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cbx4p\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:35:26.554099 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.554058 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:35:26.987299 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.987273 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p"] Nov 02 23:35:26.988780 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:35:26.988741 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f195682_c9a3_4e14_994c_5f795c95f41a.slice/crio-a53bfa778de53b1787d4dc260e8be7f7082e69b354d107f9d331ffc7c8e16805 WatchSource:0}: Error finding container a53bfa778de53b1787d4dc260e8be7f7082e69b354d107f9d331ffc7c8e16805: Status 404 returned error can't find the container with id a53bfa778de53b1787d4dc260e8be7f7082e69b354d107f9d331ffc7c8e16805 Nov 02 23:35:26.991457 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.991419 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:35:26.991610 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:26.991532 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:35:27.312519 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:27.312448 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" event={"ID":"1f195682-c9a3-4e14-994c-5f795c95f41a","Type":"ContainerStarted","Data":"e9b78df2860d36241278dcce6eb2902385145581eefa8f58b3814d77a2569a47"} Nov 02 23:35:27.312945 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:27.312528 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" event={"ID":"1f195682-c9a3-4e14-994c-5f795c95f41a","Type":"ContainerStarted","Data":"a53bfa778de53b1787d4dc260e8be7f7082e69b354d107f9d331ffc7c8e16805"} Nov 02 23:35:27.312945 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:27.312560 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:35:27.434872 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:27.434816 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" podStartSLOduration=1.434801029 podStartE2EDuration="1.434801029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:35:27.432137691 +0000 UTC m=+2802.785171063" watchObservedRunningTime="2026-11-02 23:35:27.434801029 +0000 UTC m=+2802.787834403" Nov 02 23:35:28.323461 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:28.323419 5114 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-cbx4p container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:35:28.324073 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:28.323507 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" podUID="1f195682-c9a3-4e14-994c-5f795c95f41a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:35:30.224159 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.224088 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:35:30.253543 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.253477 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-kube-api-access-kv89k\" (UniqueName: \"kubernetes.io/projected/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-kube-api-access-kv89k\") pod \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " Nov 02 23:35:30.253741 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.253574 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-token\" (UniqueName: \"kubernetes.io/projected/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-token\") pod \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " Nov 02 23:35:30.253741 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.253653 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-workload-socket\") pod \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " Nov 02 23:35:30.253741 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.253698 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-data\") pod \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " Nov 02 23:35:30.253741 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.253723 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-podinfo\") pod \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " Nov 02 23:35:30.253977 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.253746 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-workload-certs\") pod \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " Nov 02 23:35:30.253977 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.253820 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-credential-socket\") pod \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " Nov 02 23:35:30.253977 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.253930 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istiod-ca-cert\") pod \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " Nov 02 23:35:30.253977 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.253974 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-envoy\") pod \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\" (UID: \"afc01b1c-b566-4d06-bd25-c99d8b60ac3c\") " Nov 02 23:35:30.254172 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.253984 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-data" pod "afc01b1c-b566-4d06-bd25-c99d8b60ac3c" (UID: "afc01b1c-b566-4d06-bd25-c99d8b60ac3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:30.254611 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.254269 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-credential-socket" pod "afc01b1c-b566-4d06-bd25-c99d8b60ac3c" (UID: "afc01b1c-b566-4d06-bd25-c99d8b60ac3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:30.254611 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.254337 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-workload-certs" pod "afc01b1c-b566-4d06-bd25-c99d8b60ac3c" (UID: "afc01b1c-b566-4d06-bd25-c99d8b60ac3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:30.254611 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.254559 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:30.254611 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.254580 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:30.254611 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.254584 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istiod-ca-cert" pod "afc01b1c-b566-4d06-bd25-c99d8b60ac3c" (UID: "afc01b1c-b566-4d06-bd25-c99d8b60ac3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:35:30.254959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.254595 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:30.254959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.253973 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-workload-socket" pod "afc01b1c-b566-4d06-bd25-c99d8b60ac3c" (UID: "afc01b1c-b566-4d06-bd25-c99d8b60ac3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:30.256213 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.256192 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-token" pod "afc01b1c-b566-4d06-bd25-c99d8b60ac3c" (UID: "afc01b1c-b566-4d06-bd25-c99d8b60ac3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:30.256467 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.256449 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-envoy" pod "afc01b1c-b566-4d06-bd25-c99d8b60ac3c" (UID: "afc01b1c-b566-4d06-bd25-c99d8b60ac3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:30.256633 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.256614 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-kube-api-access-kv89k" pod "afc01b1c-b566-4d06-bd25-c99d8b60ac3c" (UID: "afc01b1c-b566-4d06-bd25-c99d8b60ac3c"). InnerVolumeSpecName "kube-api-access-kv89k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:30.256690 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.256615 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-podinfo" pod "afc01b1c-b566-4d06-bd25-c99d8b60ac3c" (UID: "afc01b1c-b566-4d06-bd25-c99d8b60ac3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:35:30.334335 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.334300 5114 generic.go:356] "Generic (PLEG): container finished" podID="afc01b1c-b566-4d06-bd25-c99d8b60ac3c" containerID="edc8c2aafdae525b4707f98ffb42c81facfe701d759b799e10d97058529742d0" exitCode=0 Nov 02 23:35:30.334536 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.334346 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" Nov 02 23:35:30.334536 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.334367 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" event={"ID":"afc01b1c-b566-4d06-bd25-c99d8b60ac3c","Type":"ContainerDied","Data":"edc8c2aafdae525b4707f98ffb42c81facfe701d759b799e10d97058529742d0"} Nov 02 23:35:30.334536 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.334457 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl" event={"ID":"afc01b1c-b566-4d06-bd25-c99d8b60ac3c","Type":"ContainerDied","Data":"6c93c85232a7cd71019e7e6247a52929ef63bf75ce491ac7b49045c14532e786"} Nov 02 23:35:30.334536 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.334517 5114 scope.go:122] "RemoveContainer" containerID="edc8c2aafdae525b4707f98ffb42c81facfe701d759b799e10d97058529742d0" Nov 02 23:35:30.344937 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.344920 5114 scope.go:122] "RemoveContainer" containerID="1a19aca8352227c2e610d9e58d332ad83197d6f35c51312905405c7af2c7ab00" Nov 02 23:35:30.355611 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.355501 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:30.355611 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.355534 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:30.355611 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.355548 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv89k\" (UniqueName: \"kubernetes.io/projected/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-kube-api-access-kv89k\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:30.355611 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.355563 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:30.355611 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.355576 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:30.355611 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.355588 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc01b1c-b566-4d06-bd25-c99d8b60ac3c-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:30.355897 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.355666 5114 scope.go:122] "RemoveContainer" containerID="edc8c2aafdae525b4707f98ffb42c81facfe701d759b799e10d97058529742d0" Nov 02 23:35:30.356041 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:35:30.356020 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edc8c2aafdae525b4707f98ffb42c81facfe701d759b799e10d97058529742d0\": container with ID starting with edc8c2aafdae525b4707f98ffb42c81facfe701d759b799e10d97058529742d0 not found: ID does not exist" containerID="edc8c2aafdae525b4707f98ffb42c81facfe701d759b799e10d97058529742d0" Nov 02 23:35:30.356080 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.356053 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc8c2aafdae525b4707f98ffb42c81facfe701d759b799e10d97058529742d0"} err="failed to get container status \"edc8c2aafdae525b4707f98ffb42c81facfe701d759b799e10d97058529742d0\": rpc error: code = NotFound desc = could not find container \"edc8c2aafdae525b4707f98ffb42c81facfe701d759b799e10d97058529742d0\": container with ID starting with edc8c2aafdae525b4707f98ffb42c81facfe701d759b799e10d97058529742d0 not found: ID does not exist" Nov 02 23:35:30.356080 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.356073 5114 scope.go:122] "RemoveContainer" containerID="1a19aca8352227c2e610d9e58d332ad83197d6f35c51312905405c7af2c7ab00" Nov 02 23:35:30.356352 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:35:30.356331 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a19aca8352227c2e610d9e58d332ad83197d6f35c51312905405c7af2c7ab00\": container with ID starting with 1a19aca8352227c2e610d9e58d332ad83197d6f35c51312905405c7af2c7ab00 not found: ID does not exist" containerID="1a19aca8352227c2e610d9e58d332ad83197d6f35c51312905405c7af2c7ab00" Nov 02 23:35:30.356422 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.356361 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a19aca8352227c2e610d9e58d332ad83197d6f35c51312905405c7af2c7ab00"} err="failed to get container status \"1a19aca8352227c2e610d9e58d332ad83197d6f35c51312905405c7af2c7ab00\": rpc error: code = NotFound desc = could not find container \"1a19aca8352227c2e610d9e58d332ad83197d6f35c51312905405c7af2c7ab00\": container with ID starting with 1a19aca8352227c2e610d9e58d332ad83197d6f35c51312905405c7af2c7ab00 not found: ID does not exist" Nov 02 23:35:30.508803 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.508713 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl"] Nov 02 23:35:30.514247 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.514216 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-cpnfl"] Nov 02 23:35:30.794460 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:30.794422 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afc01b1c-b566-4d06-bd25-c99d8b60ac3c" path="/var/lib/kubelet/pods/afc01b1c-b566-4d06-bd25-c99d8b60ac3c/volumes" Nov 02 23:35:31.321573 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:31.321537 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:35:33.046353 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:33.046318 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6476e8ca-fk898"] Nov 02 23:35:33.095652 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:33.095615 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6476e8ca-fk898"] Nov 02 23:35:34.794117 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:34.794083 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28b616d3-66da-4466-8880-de11323f6ad9" path="/var/lib/kubelet/pods/28b616d3-66da-4466-8880-de11323f6ad9/volumes" Nov 02 23:35:37.586353 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:35:37.586314 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:35:43.050499 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:43.050412 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ea7652b-wk5jw"] Nov 02 23:35:43.099847 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:43.099809 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ea7652b-wk5jw"] Nov 02 23:35:44.796224 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:35:44.796176 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8de33585-6958-4a92-b698-a70b1ec67165" path="/var/lib/kubelet/pods/8de33585-6958-4a92-b698-a70b1ec67165/volumes" Nov 02 23:36:02.100151 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.100100 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2"] Nov 02 23:36:02.100770 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.100396 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2" podUID="1a6b0052-2d10-47d5-ae06-95d5fc77de1f" containerName="payload-processing" containerID="cri-o://2e2f8a9fa2ca950f3f7096189ad42dec62a73a7b979d87b2d5afcea10e7e5dfe" gracePeriod=30 Nov 02 23:36:02.161022 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.160982 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h"] Nov 02 23:36:02.161229 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.161192 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h" podUID="8f7747ad-4a88-41c8-95d2-bfac505c05ca" containerName="payload-pre-processing" containerID="cri-o://0f331948c34489cd90769c6884515c5540dd27d8207c30ff068ac71b604aa9b8" gracePeriod=30 Nov 02 23:36:02.434319 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.434288 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2" Nov 02 23:36:02.464931 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.464903 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h" Nov 02 23:36:02.520021 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.519980 5114 generic.go:356] "Generic (PLEG): container finished" podID="8f7747ad-4a88-41c8-95d2-bfac505c05ca" containerID="0f331948c34489cd90769c6884515c5540dd27d8207c30ff068ac71b604aa9b8" exitCode=0 Nov 02 23:36:02.520238 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.520029 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h" Nov 02 23:36:02.520238 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.520069 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h" event={"ID":"8f7747ad-4a88-41c8-95d2-bfac505c05ca","Type":"ContainerDied","Data":"0f331948c34489cd90769c6884515c5540dd27d8207c30ff068ac71b604aa9b8"} Nov 02 23:36:02.520238 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.520170 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h" event={"ID":"8f7747ad-4a88-41c8-95d2-bfac505c05ca","Type":"ContainerDied","Data":"02b5b41a322525b9928884b14e34bcfaa1c631359e17fd8c00470531fad91143"} Nov 02 23:36:02.520238 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.520200 5114 scope.go:122] "RemoveContainer" containerID="0f331948c34489cd90769c6884515c5540dd27d8207c30ff068ac71b604aa9b8" Nov 02 23:36:02.521616 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.521585 5114 generic.go:356] "Generic (PLEG): container finished" podID="1a6b0052-2d10-47d5-ae06-95d5fc77de1f" containerID="2e2f8a9fa2ca950f3f7096189ad42dec62a73a7b979d87b2d5afcea10e7e5dfe" exitCode=0 Nov 02 23:36:02.521754 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.521626 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2" event={"ID":"1a6b0052-2d10-47d5-ae06-95d5fc77de1f","Type":"ContainerDied","Data":"2e2f8a9fa2ca950f3f7096189ad42dec62a73a7b979d87b2d5afcea10e7e5dfe"} Nov 02 23:36:02.521754 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.521680 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2" event={"ID":"1a6b0052-2d10-47d5-ae06-95d5fc77de1f","Type":"ContainerDied","Data":"7965001ac88378f2ba504aea60ab8e9d0150e89440fc19984d22b9b609bcdbee"} Nov 02 23:36:02.521754 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.521634 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2" Nov 02 23:36:02.529958 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.529933 5114 scope.go:122] "RemoveContainer" containerID="0f331948c34489cd90769c6884515c5540dd27d8207c30ff068ac71b604aa9b8" Nov 02 23:36:02.530336 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:36:02.530315 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f331948c34489cd90769c6884515c5540dd27d8207c30ff068ac71b604aa9b8\": container with ID starting with 0f331948c34489cd90769c6884515c5540dd27d8207c30ff068ac71b604aa9b8 not found: ID does not exist" containerID="0f331948c34489cd90769c6884515c5540dd27d8207c30ff068ac71b604aa9b8" Nov 02 23:36:02.530384 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.530352 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f331948c34489cd90769c6884515c5540dd27d8207c30ff068ac71b604aa9b8"} err="failed to get container status \"0f331948c34489cd90769c6884515c5540dd27d8207c30ff068ac71b604aa9b8\": rpc error: code = NotFound desc = could not find container \"0f331948c34489cd90769c6884515c5540dd27d8207c30ff068ac71b604aa9b8\": container with ID starting with 0f331948c34489cd90769c6884515c5540dd27d8207c30ff068ac71b604aa9b8 not found: ID does not exist" Nov 02 23:36:02.530384 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.530373 5114 scope.go:122] "RemoveContainer" containerID="2e2f8a9fa2ca950f3f7096189ad42dec62a73a7b979d87b2d5afcea10e7e5dfe" Nov 02 23:36:02.539848 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.539820 5114 scope.go:122] "RemoveContainer" containerID="2e2f8a9fa2ca950f3f7096189ad42dec62a73a7b979d87b2d5afcea10e7e5dfe" Nov 02 23:36:02.540195 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:36:02.540173 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2f8a9fa2ca950f3f7096189ad42dec62a73a7b979d87b2d5afcea10e7e5dfe\": container with ID starting with 2e2f8a9fa2ca950f3f7096189ad42dec62a73a7b979d87b2d5afcea10e7e5dfe not found: ID does not exist" containerID="2e2f8a9fa2ca950f3f7096189ad42dec62a73a7b979d87b2d5afcea10e7e5dfe" Nov 02 23:36:02.540249 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.540208 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2f8a9fa2ca950f3f7096189ad42dec62a73a7b979d87b2d5afcea10e7e5dfe"} err="failed to get container status \"2e2f8a9fa2ca950f3f7096189ad42dec62a73a7b979d87b2d5afcea10e7e5dfe\": rpc error: code = NotFound desc = could not find container \"2e2f8a9fa2ca950f3f7096189ad42dec62a73a7b979d87b2d5afcea10e7e5dfe\": container with ID starting with 2e2f8a9fa2ca950f3f7096189ad42dec62a73a7b979d87b2d5afcea10e7e5dfe not found: ID does not exist" Nov 02 23:36:02.546686 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.546657 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a6b0052-2d10-47d5-ae06-95d5fc77de1f-kube-api-access-gqlnf\" (UniqueName: \"kubernetes.io/projected/1a6b0052-2d10-47d5-ae06-95d5fc77de1f-kube-api-access-gqlnf\") pod \"1a6b0052-2d10-47d5-ae06-95d5fc77de1f\" (UID: \"1a6b0052-2d10-47d5-ae06-95d5fc77de1f\") " Nov 02 23:36:02.546869 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.546732 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f7747ad-4a88-41c8-95d2-bfac505c05ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f7747ad-4a88-41c8-95d2-bfac505c05ca-plugins-config-volume\") pod \"8f7747ad-4a88-41c8-95d2-bfac505c05ca\" (UID: \"8f7747ad-4a88-41c8-95d2-bfac505c05ca\") " Nov 02 23:36:02.546932 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.546866 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a6b0052-2d10-47d5-ae06-95d5fc77de1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6b0052-2d10-47d5-ae06-95d5fc77de1f-plugins-config-volume\") pod \"1a6b0052-2d10-47d5-ae06-95d5fc77de1f\" (UID: \"1a6b0052-2d10-47d5-ae06-95d5fc77de1f\") " Nov 02 23:36:02.546985 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.546954 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f7747ad-4a88-41c8-95d2-bfac505c05ca-kube-api-access-qwdcd\" (UniqueName: \"kubernetes.io/projected/8f7747ad-4a88-41c8-95d2-bfac505c05ca-kube-api-access-qwdcd\") pod \"8f7747ad-4a88-41c8-95d2-bfac505c05ca\" (UID: \"8f7747ad-4a88-41c8-95d2-bfac505c05ca\") " Nov 02 23:36:02.547235 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.547205 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f7747ad-4a88-41c8-95d2-bfac505c05ca-plugins-config-volume" pod "8f7747ad-4a88-41c8-95d2-bfac505c05ca" (UID: "8f7747ad-4a88-41c8-95d2-bfac505c05ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:36:02.547373 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.547286 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a6b0052-2d10-47d5-ae06-95d5fc77de1f-plugins-config-volume" pod "1a6b0052-2d10-47d5-ae06-95d5fc77de1f" (UID: "1a6b0052-2d10-47d5-ae06-95d5fc77de1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:36:02.547578 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.547558 5114 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6b0052-2d10-47d5-ae06-95d5fc77de1f-plugins-config-volume\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:36:02.547653 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.547579 5114 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f7747ad-4a88-41c8-95d2-bfac505c05ca-plugins-config-volume\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:36:02.549329 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.549293 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7747ad-4a88-41c8-95d2-bfac505c05ca-kube-api-access-qwdcd" pod "8f7747ad-4a88-41c8-95d2-bfac505c05ca" (UID: "8f7747ad-4a88-41c8-95d2-bfac505c05ca"). InnerVolumeSpecName "kube-api-access-qwdcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:36:02.549473 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.549393 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a6b0052-2d10-47d5-ae06-95d5fc77de1f-kube-api-access-gqlnf" pod "1a6b0052-2d10-47d5-ae06-95d5fc77de1f" (UID: "1a6b0052-2d10-47d5-ae06-95d5fc77de1f"). InnerVolumeSpecName "kube-api-access-gqlnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:36:02.648568 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.648447 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwdcd\" (UniqueName: \"kubernetes.io/projected/8f7747ad-4a88-41c8-95d2-bfac505c05ca-kube-api-access-qwdcd\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:36:02.648568 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.648512 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqlnf\" (UniqueName: \"kubernetes.io/projected/1a6b0052-2d10-47d5-ae06-95d5fc77de1f-kube-api-access-gqlnf\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:36:02.936693 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.936591 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h"] Nov 02 23:36:02.940522 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:02.940472 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-965487dc5-rc85h"] Nov 02 23:36:03.049441 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:03.049405 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2"] Nov 02 23:36:03.051574 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:03.051548 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2279ad-59c5bb6766-cprf2"] Nov 02 23:36:04.793599 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:04.793564 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a6b0052-2d10-47d5-ae06-95d5fc77de1f" path="/var/lib/kubelet/pods/1a6b0052-2d10-47d5-ae06-95d5fc77de1f/volumes" Nov 02 23:36:04.794124 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:04.794105 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f7747ad-4a88-41c8-95d2-bfac505c05ca" path="/var/lib/kubelet/pods/8f7747ad-4a88-41c8-95d2-bfac505c05ca/volumes" Nov 02 23:36:07.586459 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:36:07.586416 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:36:37.585801 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:36:37.585756 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:36:40.775456 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:40.775422 5114 generic.go:356] "Generic (PLEG): container finished" podID="55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" containerID="3800a86b7c6190d0fb87f7bef48260406279358468ecad3354f8cafaaf7286bb" exitCode=0 Nov 02 23:36:40.775963 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:40.775512 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" event={"ID":"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046","Type":"ContainerDied","Data":"3800a86b7c6190d0fb87f7bef48260406279358468ecad3354f8cafaaf7286bb"} Nov 02 23:36:40.776010 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:40.775973 5114 scope.go:122] "RemoveContainer" containerID="3800a86b7c6190d0fb87f7bef48260406279358468ecad3354f8cafaaf7286bb" Nov 02 23:36:40.777874 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:40.777837 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:36:40.777977 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:40.777936 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:36:40.778015 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:40.777986 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:36:41.783532 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:41.783475 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" event={"ID":"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046","Type":"ContainerStarted","Data":"e7bfd35221ff1a530f36ca03d0067cabf49c05ce34598bc44414d286ba20cd43"} Nov 02 23:36:41.813178 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:41.813143 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:36:41.814250 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:41.814219 5114 patch_prober.go:29] interesting pod/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 23:36:41.814400 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:41.814291 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" podUID="55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 23:36:42.830995 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:42.830949 5114 patch_prober.go:29] interesting pod/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 23:36:42.831446 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:42.831023 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" podUID="55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 23:36:43.819508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:43.818903 5114 patch_prober.go:29] interesting pod/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 23:36:43.819508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:43.818970 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" podUID="55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 23:36:44.828716 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:44.828199 5114 patch_prober.go:29] interesting pod/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 23:36:44.828716 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:44.828281 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" podUID="55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 23:36:45.140306 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:45.140217 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:36:45.433759 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:45.433667 5114 scope.go:122] "RemoveContainer" containerID="4a1f1f4bbc4b85802ebd199e480a7f37df08bd3df9470182c1928f52132a738b" Nov 02 23:36:45.443851 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:45.443825 5114 scope.go:122] "RemoveContainer" containerID="e46493c075dfd9155b5b21196b9f4eb6f9ba20131cec2c9f7d68308037deb27a" Nov 02 23:36:45.826648 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:45.826604 5114 patch_prober.go:29] interesting pod/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 23:36:45.826961 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:45.826927 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" podUID="55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 23:36:46.826973 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:46.826933 5114 patch_prober.go:29] interesting pod/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 23:36:46.827656 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:46.827620 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" podUID="55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 23:36:47.818558 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:47.818523 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:36:47.819714 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:36:47.819689 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 02 23:37:07.583526 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:37:07.583460 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:37:12.225867 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.225815 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws"] Nov 02 23:37:12.226448 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.226427 5114 cpu_manager.go:414] "containerMap: removing container" podUID="1a6b0052-2d10-47d5-ae06-95d5fc77de1f" containerName="payload-processing" Nov 02 23:37:12.226525 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.226451 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a6b0052-2d10-47d5-ae06-95d5fc77de1f" containerName="payload-processing" Nov 02 23:37:12.226525 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.226469 5114 cpu_manager.go:414] "containerMap: removing container" podUID="afc01b1c-b566-4d06-bd25-c99d8b60ac3c" containerName="istio-proxy" Nov 02 23:37:12.226525 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.226475 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afc01b1c-b566-4d06-bd25-c99d8b60ac3c" containerName="istio-proxy" Nov 02 23:37:12.226525 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.226524 5114 cpu_manager.go:414] "containerMap: removing container" podUID="8f7747ad-4a88-41c8-95d2-bfac505c05ca" containerName="payload-pre-processing" Nov 02 23:37:12.226658 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.226530 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f7747ad-4a88-41c8-95d2-bfac505c05ca" containerName="payload-pre-processing" Nov 02 23:37:12.226658 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.226546 5114 cpu_manager.go:414] "containerMap: removing container" podUID="afc01b1c-b566-4d06-bd25-c99d8b60ac3c" containerName="istio-proxy" Nov 02 23:37:12.226658 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.226551 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afc01b1c-b566-4d06-bd25-c99d8b60ac3c" containerName="istio-proxy" Nov 02 23:37:12.226658 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.226617 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="afc01b1c-b566-4d06-bd25-c99d8b60ac3c" containerName="istio-proxy" Nov 02 23:37:12.226658 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.226626 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f7747ad-4a88-41c8-95d2-bfac505c05ca" containerName="payload-pre-processing" Nov 02 23:37:12.226658 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.226637 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a6b0052-2d10-47d5-ae06-95d5fc77de1f" containerName="payload-processing" Nov 02 23:37:12.231376 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.231354 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:37:12.289743 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.289691 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-tmp-dir\") pod \"praxis-route-94e3ae-kserve-659d6895c5-4zjws\" (UID: \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\") " pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:37:12.289743 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.289745 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7jm6\" (UniqueName: \"kubernetes.io/projected/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-kube-api-access-v7jm6\") pod \"praxis-route-94e3ae-kserve-659d6895c5-4zjws\" (UID: \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\") " pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:37:12.289994 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.289937 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-tls-certs\") pod \"praxis-route-94e3ae-kserve-659d6895c5-4zjws\" (UID: \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\") " pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:37:12.289994 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.289987 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-home\") pod \"praxis-route-94e3ae-kserve-659d6895c5-4zjws\" (UID: \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\") " pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:37:12.290110 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.290020 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-dshm\") pod \"praxis-route-94e3ae-kserve-659d6895c5-4zjws\" (UID: \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\") " pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:37:12.290110 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.290094 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-model-cache\") pod \"praxis-route-94e3ae-kserve-659d6895c5-4zjws\" (UID: \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\") " pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:37:12.334872 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.334837 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-94e3ae\"/\"kube-root-ca.crt\"" Nov 02 23:37:12.334872 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.334842 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-94e3ae\"/\"openshift-service-ca.crt\"" Nov 02 23:37:12.335520 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.335463 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-94e3ae\"/\"default-dockercfg-8fjwx\"" Nov 02 23:37:12.335646 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.335578 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-94e3ae\"/\"praxis-route-94e3ae-kserve-self-signed-certs\"" Nov 02 23:37:12.338381 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.338351 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws"] Nov 02 23:37:12.391057 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.391020 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-tls-certs\") pod \"praxis-route-94e3ae-kserve-659d6895c5-4zjws\" (UID: \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\") " pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:37:12.391057 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.391063 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-home\") pod \"praxis-route-94e3ae-kserve-659d6895c5-4zjws\" (UID: \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\") " pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:37:12.391447 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.391081 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-dshm\") pod \"praxis-route-94e3ae-kserve-659d6895c5-4zjws\" (UID: \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\") " pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:37:12.391447 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.391124 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-model-cache\") pod \"praxis-route-94e3ae-kserve-659d6895c5-4zjws\" (UID: \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\") " pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:37:12.391447 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.391205 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-tmp-dir\") pod \"praxis-route-94e3ae-kserve-659d6895c5-4zjws\" (UID: \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\") " pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:37:12.391447 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.391222 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7jm6\" (UniqueName: \"kubernetes.io/projected/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-kube-api-access-v7jm6\") pod \"praxis-route-94e3ae-kserve-659d6895c5-4zjws\" (UID: \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\") " pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:37:12.391871 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.391648 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-model-cache\") pod \"praxis-route-94e3ae-kserve-659d6895c5-4zjws\" (UID: \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\") " pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:37:12.391871 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.391818 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-home\") pod \"praxis-route-94e3ae-kserve-659d6895c5-4zjws\" (UID: \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\") " pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:37:12.391871 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.391835 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-tmp-dir\") pod \"praxis-route-94e3ae-kserve-659d6895c5-4zjws\" (UID: \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\") " pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:37:12.393784 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.393762 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-dshm\") pod \"praxis-route-94e3ae-kserve-659d6895c5-4zjws\" (UID: \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\") " pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:37:12.395047 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.395021 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-tls-certs\") pod \"praxis-route-94e3ae-kserve-659d6895c5-4zjws\" (UID: \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\") " pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:37:12.452584 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.452540 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7jm6\" (UniqueName: \"kubernetes.io/projected/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-kube-api-access-v7jm6\") pod \"praxis-route-94e3ae-kserve-659d6895c5-4zjws\" (UID: \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\") " pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:37:12.543123 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.543078 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:37:12.928821 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.928790 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws"] Nov 02 23:37:12.930554 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:37:12.930523 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d314a4e_a7f3_41b6_b2e5_671bc57d9c62.slice/crio-5970272b15ebb2cda7a233822e1549dede83069226acd6190e5f37738237f1a9 WatchSource:0}: Error finding container 5970272b15ebb2cda7a233822e1549dede83069226acd6190e5f37738237f1a9: Status 404 returned error can't find the container with id 5970272b15ebb2cda7a233822e1549dede83069226acd6190e5f37738237f1a9 Nov 02 23:37:12.976956 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:12.976916 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" event={"ID":"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62","Type":"ContainerStarted","Data":"5970272b15ebb2cda7a233822e1549dede83069226acd6190e5f37738237f1a9"} Nov 02 23:37:13.984857 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:13.984802 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" event={"ID":"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62","Type":"ContainerStarted","Data":"5505337994c33da7ba1434295c6a1f285d68d351978390ec82445c3737865332"} Nov 02 23:37:14.100748 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:14.100681 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" podStartSLOduration=2.100663167 podStartE2EDuration="2.100663167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:37:14.099693977 +0000 UTC m=+2909.452727352" watchObservedRunningTime="2026-11-02 23:37:14.100663167 +0000 UTC m=+2909.453696541" Nov 02 23:37:14.543948 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:14.543915 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:37:22.544139 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:22.544099 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:37:22.557216 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:22.557178 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:37:23.049148 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:23.049118 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:37:26.058819 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:26.058774 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51110ea5-v7sdp"] Nov 02 23:37:26.107732 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:26.107683 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51110ea5-v7sdp"] Nov 02 23:37:26.793760 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:26.793718 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f296a3b6-d7ef-477a-8221-b24528a9b8f4" path="/var/lib/kubelet/pods/f296a3b6-d7ef-477a-8221-b24528a9b8f4/volumes" Nov 02 23:37:37.584426 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:37:37.584384 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:37:49.047350 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:49.047311 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-567e11bd-4rbsn"] Nov 02 23:37:49.096652 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:49.096619 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-567e11bd-4rbsn"] Nov 02 23:37:50.793468 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:37:50.793423 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1334b934-4cbb-405d-a48c-125af5e6b07d" path="/var/lib/kubelet/pods/1334b934-4cbb-405d-a48c-125af5e6b07d/volumes" Nov 02 23:38:06.049787 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:06.049751 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87d506e2-mgpn8"] Nov 02 23:38:06.099138 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:06.099103 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87d506e2-mgpn8"] Nov 02 23:38:06.793022 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:06.792980 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8" path="/var/lib/kubelet/pods/80d8cf2e-9b7d-42b0-a7f8-3e216b885ce8/volumes" Nov 02 23:38:07.586015 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:38:07.585974 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:38:13.264325 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.264282 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x"] Nov 02 23:38:13.265058 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.265034 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="afc01b1c-b566-4d06-bd25-c99d8b60ac3c" containerName="istio-proxy" Nov 02 23:38:13.268126 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.268105 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:38:13.370570 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.370534 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f46b61\"/\"kube-root-ca.crt\"" Nov 02 23:38:13.370769 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.370604 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f46b61\"/\"default-dockercfg-9thd6\"" Nov 02 23:38:13.370769 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.370624 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f46b61\"/\"auth-test-model-f46b61-kserve-self-signed-certs\"" Nov 02 23:38:13.370769 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.370696 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f46b61\"/\"openshift-service-ca.crt\"" Nov 02 23:38:13.373080 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.373056 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x"] Nov 02 23:38:13.393610 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.393571 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-home\") pod \"auth-test-model-f46b61-kserve-c69cf7585-vh62x\" (UID: \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\") " pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:38:13.393817 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.393689 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-dshm\") pod \"auth-test-model-f46b61-kserve-c69cf7585-vh62x\" (UID: \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\") " pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:38:13.393817 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.393772 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-model-cache\") pod \"auth-test-model-f46b61-kserve-c69cf7585-vh62x\" (UID: \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\") " pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:38:13.394001 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.393898 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-tmp-dir\") pod \"auth-test-model-f46b61-kserve-c69cf7585-vh62x\" (UID: \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\") " pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:38:13.394001 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.393939 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qjbr\" (UniqueName: \"kubernetes.io/projected/7a521ccd-9afc-4dcb-af5e-de54dabb7340-kube-api-access-9qjbr\") pod \"auth-test-model-f46b61-kserve-c69cf7585-vh62x\" (UID: \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\") " pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:38:13.394124 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.394007 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a521ccd-9afc-4dcb-af5e-de54dabb7340-tls-certs\") pod \"auth-test-model-f46b61-kserve-c69cf7585-vh62x\" (UID: \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\") " pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:38:13.495159 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.495109 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a521ccd-9afc-4dcb-af5e-de54dabb7340-tls-certs\") pod \"auth-test-model-f46b61-kserve-c69cf7585-vh62x\" (UID: \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\") " pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:38:13.495385 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.495175 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-home\") pod \"auth-test-model-f46b61-kserve-c69cf7585-vh62x\" (UID: \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\") " pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:38:13.495385 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.495238 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-dshm\") pod \"auth-test-model-f46b61-kserve-c69cf7585-vh62x\" (UID: \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\") " pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:38:13.495385 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.495331 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-model-cache\") pod \"auth-test-model-f46b61-kserve-c69cf7585-vh62x\" (UID: \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\") " pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:38:13.495604 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.495449 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-tmp-dir\") pod \"auth-test-model-f46b61-kserve-c69cf7585-vh62x\" (UID: \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\") " pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:38:13.495604 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.495520 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qjbr\" (UniqueName: \"kubernetes.io/projected/7a521ccd-9afc-4dcb-af5e-de54dabb7340-kube-api-access-9qjbr\") pod \"auth-test-model-f46b61-kserve-c69cf7585-vh62x\" (UID: \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\") " pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:38:13.495773 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.495743 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-home\") pod \"auth-test-model-f46b61-kserve-c69cf7585-vh62x\" (UID: \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\") " pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:38:13.495925 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.495810 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-model-cache\") pod \"auth-test-model-f46b61-kserve-c69cf7585-vh62x\" (UID: \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\") " pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:38:13.495998 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.495944 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-tmp-dir\") pod \"auth-test-model-f46b61-kserve-c69cf7585-vh62x\" (UID: \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\") " pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:38:13.497679 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.497657 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-dshm\") pod \"auth-test-model-f46b61-kserve-c69cf7585-vh62x\" (UID: \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\") " pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:38:13.497942 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.497923 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a521ccd-9afc-4dcb-af5e-de54dabb7340-tls-certs\") pod \"auth-test-model-f46b61-kserve-c69cf7585-vh62x\" (UID: \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\") " pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:38:13.556380 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.556344 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qjbr\" (UniqueName: \"kubernetes.io/projected/7a521ccd-9afc-4dcb-af5e-de54dabb7340-kube-api-access-9qjbr\") pod \"auth-test-model-f46b61-kserve-c69cf7585-vh62x\" (UID: \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\") " pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:38:13.580769 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.580713 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:38:13.966469 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:13.966441 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x"] Nov 02 23:38:13.968038 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:38:13.968002 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a521ccd_9afc_4dcb_af5e_de54dabb7340.slice/crio-b9b79e46bcbaf512346ac648356b29b0077aa1da6a886aa4e803dc639dc350e6 WatchSource:0}: Error finding container b9b79e46bcbaf512346ac648356b29b0077aa1da6a886aa4e803dc639dc350e6: Status 404 returned error can't find the container with id b9b79e46bcbaf512346ac648356b29b0077aa1da6a886aa4e803dc639dc350e6 Nov 02 23:38:14.327613 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:14.327569 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" event={"ID":"7a521ccd-9afc-4dcb-af5e-de54dabb7340","Type":"ContainerStarted","Data":"0c5963ce5657d78689de0e67a689f339ace1270e6cddf05ffb36b4413055663f"} Nov 02 23:38:14.328131 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:14.327625 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" event={"ID":"7a521ccd-9afc-4dcb-af5e-de54dabb7340","Type":"ContainerStarted","Data":"b9b79e46bcbaf512346ac648356b29b0077aa1da6a886aa4e803dc639dc350e6"} Nov 02 23:38:14.446247 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:14.446186 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" podStartSLOduration=1.4461718860000001 podStartE2EDuration="1.446171886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:38:14.443460243 +0000 UTC m=+2969.796493618" watchObservedRunningTime="2026-11-02 23:38:14.446171886 +0000 UTC m=+2969.799205260" Nov 02 23:38:14.581145 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:14.581058 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:38:23.581068 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:23.581022 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:38:23.594103 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:23.594069 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:38:24.393954 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:24.393875 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:38:29.708331 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:29.708297 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj"] Nov 02 23:38:29.708835 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:29.708555 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" podUID="a9840bb1-3a88-4d36-8077-1facecc81158" containerName="istio-proxy" containerID="cri-o://69b2049e9a94810b8fb40bf3935951ae1d1360195580d06fc89e68ff40350050" gracePeriod=30 Nov 02 23:38:29.758178 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:29.758141 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6"] Nov 02 23:38:29.758392 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:29.758355 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" podUID="5693f409-a9e5-487a-885a-c254f8ef63cd" containerName="istio-proxy" containerID="cri-o://4d7874c4db39bdfdf031f1b47383a35c339ba4aa32f081cf9b9c9be8b46f11e3" gracePeriod=30 Nov 02 23:38:35.033776 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.033746 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:38:35.069237 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.069206 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:38:35.170206 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.170166 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-envoy\") pod \"5693f409-a9e5-487a-885a-c254f8ef63cd\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " Nov 02 23:38:35.170410 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.170216 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-istio-data\") pod \"a9840bb1-3a88-4d36-8077-1facecc81158\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " Nov 02 23:38:35.170410 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.170252 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-workload-socket\") pod \"a9840bb1-3a88-4d36-8077-1facecc81158\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " Nov 02 23:38:35.170410 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.170313 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9840bb1-3a88-4d36-8077-1facecc81158-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9840bb1-3a88-4d36-8077-1facecc81158-istio-podinfo\") pod \"a9840bb1-3a88-4d36-8077-1facecc81158\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " Nov 02 23:38:35.170410 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.170364 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5693f409-a9e5-487a-885a-c254f8ef63cd-kube-api-access-5qk42\" (UniqueName: \"kubernetes.io/projected/5693f409-a9e5-487a-885a-c254f8ef63cd-kube-api-access-5qk42\") pod \"5693f409-a9e5-487a-885a-c254f8ef63cd\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " Nov 02 23:38:35.170410 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.170397 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-token\" (UniqueName: \"kubernetes.io/projected/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-token\") pod \"5693f409-a9e5-487a-885a-c254f8ef63cd\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " Nov 02 23:38:35.170723 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.170427 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-istio-envoy\") pod \"a9840bb1-3a88-4d36-8077-1facecc81158\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " Nov 02 23:38:35.170723 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.170539 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-istio-data" pod "a9840bb1-3a88-4d36-8077-1facecc81158" (UID: "a9840bb1-3a88-4d36-8077-1facecc81158"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:35.170723 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.170545 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-workload-certs\") pod \"5693f409-a9e5-487a-885a-c254f8ef63cd\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " Nov 02 23:38:35.170723 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.170598 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5693f409-a9e5-487a-885a-c254f8ef63cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5693f409-a9e5-487a-885a-c254f8ef63cd-istiod-ca-cert\") pod \"5693f409-a9e5-487a-885a-c254f8ef63cd\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " Nov 02 23:38:35.170723 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.170629 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9840bb1-3a88-4d36-8077-1facecc81158-istio-token\" (UniqueName: \"kubernetes.io/projected/a9840bb1-3a88-4d36-8077-1facecc81158-istio-token\") pod \"a9840bb1-3a88-4d36-8077-1facecc81158\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " Nov 02 23:38:35.170723 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.170659 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-credential-socket\") pod \"5693f409-a9e5-487a-885a-c254f8ef63cd\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " Nov 02 23:38:35.170723 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.170676 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9840bb1-3a88-4d36-8077-1facecc81158-kube-api-access-m8v8q\" (UniqueName: \"kubernetes.io/projected/a9840bb1-3a88-4d36-8077-1facecc81158-kube-api-access-m8v8q\") pod \"a9840bb1-3a88-4d36-8077-1facecc81158\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " Nov 02 23:38:35.171108 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.170764 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-data\") pod \"5693f409-a9e5-487a-885a-c254f8ef63cd\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " Nov 02 23:38:35.171108 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.170788 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-workload-certs" pod "5693f409-a9e5-487a-885a-c254f8ef63cd" (UID: "5693f409-a9e5-487a-885a-c254f8ef63cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:35.171108 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.170814 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-credential-socket\") pod \"a9840bb1-3a88-4d36-8077-1facecc81158\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " Nov 02 23:38:35.171108 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.170866 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9840bb1-3a88-4d36-8077-1facecc81158-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9840bb1-3a88-4d36-8077-1facecc81158-istiod-ca-cert\") pod \"a9840bb1-3a88-4d36-8077-1facecc81158\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " Nov 02 23:38:35.171108 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.170912 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-workload-socket\") pod \"5693f409-a9e5-487a-885a-c254f8ef63cd\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " Nov 02 23:38:35.171108 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.171002 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-podinfo\") pod \"5693f409-a9e5-487a-885a-c254f8ef63cd\" (UID: \"5693f409-a9e5-487a-885a-c254f8ef63cd\") " Nov 02 23:38:35.171108 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.171013 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-credential-socket" pod "5693f409-a9e5-487a-885a-c254f8ef63cd" (UID: "5693f409-a9e5-487a-885a-c254f8ef63cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:35.171108 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.171046 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-workload-certs\") pod \"a9840bb1-3a88-4d36-8077-1facecc81158\" (UID: \"a9840bb1-3a88-4d36-8077-1facecc81158\") " Nov 02 23:38:35.171568 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.171187 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5693f409-a9e5-487a-885a-c254f8ef63cd-istiod-ca-cert" pod "5693f409-a9e5-487a-885a-c254f8ef63cd" (UID: "5693f409-a9e5-487a-885a-c254f8ef63cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:38:35.171568 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.171384 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-workload-socket" pod "a9840bb1-3a88-4d36-8077-1facecc81158" (UID: "a9840bb1-3a88-4d36-8077-1facecc81158"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:35.171568 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.171437 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-credential-socket" pod "a9840bb1-3a88-4d36-8077-1facecc81158" (UID: "a9840bb1-3a88-4d36-8077-1facecc81158"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:35.171899 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.171639 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-workload-socket" pod "5693f409-a9e5-487a-885a-c254f8ef63cd" (UID: "5693f409-a9e5-487a-885a-c254f8ef63cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:35.171899 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.171748 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-workload-certs" pod "a9840bb1-3a88-4d36-8077-1facecc81158" (UID: "a9840bb1-3a88-4d36-8077-1facecc81158"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:35.172015 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.171958 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-data" pod "5693f409-a9e5-487a-885a-c254f8ef63cd" (UID: "5693f409-a9e5-487a-885a-c254f8ef63cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:35.172015 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.171978 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9840bb1-3a88-4d36-8077-1facecc81158-istiod-ca-cert" pod "a9840bb1-3a88-4d36-8077-1facecc81158" (UID: "a9840bb1-3a88-4d36-8077-1facecc81158"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:38:35.172340 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.172205 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.172340 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.172229 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.172340 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.172244 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.172340 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.172260 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5693f409-a9e5-487a-885a-c254f8ef63cd-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.172340 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.172276 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.172340 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.172290 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.172340 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.172304 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.172340 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.172319 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9840bb1-3a88-4d36-8077-1facecc81158-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.172340 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.172332 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.172340 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.172346 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.174039 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.174001 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-envoy" pod "5693f409-a9e5-487a-885a-c254f8ef63cd" (UID: "5693f409-a9e5-487a-885a-c254f8ef63cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:35.174200 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.174185 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-podinfo" pod "5693f409-a9e5-487a-885a-c254f8ef63cd" (UID: "5693f409-a9e5-487a-885a-c254f8ef63cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:38:35.174515 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.174475 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9840bb1-3a88-4d36-8077-1facecc81158-istio-token" pod "a9840bb1-3a88-4d36-8077-1facecc81158" (UID: "a9840bb1-3a88-4d36-8077-1facecc81158"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:38:35.174619 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.174530 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5693f409-a9e5-487a-885a-c254f8ef63cd-kube-api-access-5qk42" pod "5693f409-a9e5-487a-885a-c254f8ef63cd" (UID: "5693f409-a9e5-487a-885a-c254f8ef63cd"). InnerVolumeSpecName "kube-api-access-5qk42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:38:35.174619 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.174529 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-token" pod "5693f409-a9e5-487a-885a-c254f8ef63cd" (UID: "5693f409-a9e5-487a-885a-c254f8ef63cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:38:35.174724 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.174705 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9840bb1-3a88-4d36-8077-1facecc81158-istio-podinfo" pod "a9840bb1-3a88-4d36-8077-1facecc81158" (UID: "a9840bb1-3a88-4d36-8077-1facecc81158"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:38:35.174844 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.174827 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-istio-envoy" pod "a9840bb1-3a88-4d36-8077-1facecc81158" (UID: "a9840bb1-3a88-4d36-8077-1facecc81158"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:35.175263 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.175247 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9840bb1-3a88-4d36-8077-1facecc81158-kube-api-access-m8v8q" pod "a9840bb1-3a88-4d36-8077-1facecc81158" (UID: "a9840bb1-3a88-4d36-8077-1facecc81158"). InnerVolumeSpecName "kube-api-access-m8v8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:38:35.273904 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.273809 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8v8q\" (UniqueName: \"kubernetes.io/projected/a9840bb1-3a88-4d36-8077-1facecc81158-kube-api-access-m8v8q\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.273904 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.273844 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.273904 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.273854 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.273904 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.273863 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9840bb1-3a88-4d36-8077-1facecc81158-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.273904 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.273872 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qk42\" (UniqueName: \"kubernetes.io/projected/5693f409-a9e5-487a-885a-c254f8ef63cd-kube-api-access-5qk42\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.273904 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.273880 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5693f409-a9e5-487a-885a-c254f8ef63cd-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.273904 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.273888 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9840bb1-3a88-4d36-8077-1facecc81158-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.273904 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.273896 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9840bb1-3a88-4d36-8077-1facecc81158-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.451075 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.451039 5114 generic.go:356] "Generic (PLEG): container finished" podID="5693f409-a9e5-487a-885a-c254f8ef63cd" containerID="4d7874c4db39bdfdf031f1b47383a35c339ba4aa32f081cf9b9c9be8b46f11e3" exitCode=0 Nov 02 23:38:35.451266 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.451084 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" Nov 02 23:38:35.451266 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.451116 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" event={"ID":"5693f409-a9e5-487a-885a-c254f8ef63cd","Type":"ContainerDied","Data":"4d7874c4db39bdfdf031f1b47383a35c339ba4aa32f081cf9b9c9be8b46f11e3"} Nov 02 23:38:35.451266 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.451175 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6" event={"ID":"5693f409-a9e5-487a-885a-c254f8ef63cd","Type":"ContainerDied","Data":"529274da6bbeeb977a4e567ad30b054c36baf9f8947f384ab0c83f91d2b92b52"} Nov 02 23:38:35.451266 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.451200 5114 scope.go:122] "RemoveContainer" containerID="4d7874c4db39bdfdf031f1b47383a35c339ba4aa32f081cf9b9c9be8b46f11e3" Nov 02 23:38:35.452953 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.452927 5114 generic.go:356] "Generic (PLEG): container finished" podID="a9840bb1-3a88-4d36-8077-1facecc81158" containerID="69b2049e9a94810b8fb40bf3935951ae1d1360195580d06fc89e68ff40350050" exitCode=0 Nov 02 23:38:35.453082 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.453053 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" Nov 02 23:38:35.453082 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.453063 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" event={"ID":"a9840bb1-3a88-4d36-8077-1facecc81158","Type":"ContainerDied","Data":"69b2049e9a94810b8fb40bf3935951ae1d1360195580d06fc89e68ff40350050"} Nov 02 23:38:35.453172 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.453101 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj" event={"ID":"a9840bb1-3a88-4d36-8077-1facecc81158","Type":"ContainerDied","Data":"53cb9777d5822f9b2a4f787b07fca63dad529f72005dfe8ce0250a22eb265341"} Nov 02 23:38:35.462526 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.462505 5114 scope.go:122] "RemoveContainer" containerID="4d7874c4db39bdfdf031f1b47383a35c339ba4aa32f081cf9b9c9be8b46f11e3" Nov 02 23:38:35.462900 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:38:35.462866 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d7874c4db39bdfdf031f1b47383a35c339ba4aa32f081cf9b9c9be8b46f11e3\": container with ID starting with 4d7874c4db39bdfdf031f1b47383a35c339ba4aa32f081cf9b9c9be8b46f11e3 not found: ID does not exist" containerID="4d7874c4db39bdfdf031f1b47383a35c339ba4aa32f081cf9b9c9be8b46f11e3" Nov 02 23:38:35.462994 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.462903 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7874c4db39bdfdf031f1b47383a35c339ba4aa32f081cf9b9c9be8b46f11e3"} err="failed to get container status \"4d7874c4db39bdfdf031f1b47383a35c339ba4aa32f081cf9b9c9be8b46f11e3\": rpc error: code = NotFound desc = could not find container \"4d7874c4db39bdfdf031f1b47383a35c339ba4aa32f081cf9b9c9be8b46f11e3\": container with ID starting with 4d7874c4db39bdfdf031f1b47383a35c339ba4aa32f081cf9b9c9be8b46f11e3 not found: ID does not exist" Nov 02 23:38:35.462994 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.462921 5114 scope.go:122] "RemoveContainer" containerID="69b2049e9a94810b8fb40bf3935951ae1d1360195580d06fc89e68ff40350050" Nov 02 23:38:35.472669 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.472643 5114 scope.go:122] "RemoveContainer" containerID="69b2049e9a94810b8fb40bf3935951ae1d1360195580d06fc89e68ff40350050" Nov 02 23:38:35.473023 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:38:35.472999 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69b2049e9a94810b8fb40bf3935951ae1d1360195580d06fc89e68ff40350050\": container with ID starting with 69b2049e9a94810b8fb40bf3935951ae1d1360195580d06fc89e68ff40350050 not found: ID does not exist" containerID="69b2049e9a94810b8fb40bf3935951ae1d1360195580d06fc89e68ff40350050" Nov 02 23:38:35.473091 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.473037 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69b2049e9a94810b8fb40bf3935951ae1d1360195580d06fc89e68ff40350050"} err="failed to get container status \"69b2049e9a94810b8fb40bf3935951ae1d1360195580d06fc89e68ff40350050\": rpc error: code = NotFound desc = could not find container \"69b2049e9a94810b8fb40bf3935951ae1d1360195580d06fc89e68ff40350050\": container with ID starting with 69b2049e9a94810b8fb40bf3935951ae1d1360195580d06fc89e68ff40350050 not found: ID does not exist" Nov 02 23:38:35.615379 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.615344 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6"] Nov 02 23:38:35.619236 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.619209 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-4slr6"] Nov 02 23:38:35.728297 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.728257 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj"] Nov 02 23:38:35.733136 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:35.733099 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-8qhbj"] Nov 02 23:38:36.793545 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:36.793510 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5693f409-a9e5-487a-885a-c254f8ef63cd" path="/var/lib/kubelet/pods/5693f409-a9e5-487a-885a-c254f8ef63cd/volumes" Nov 02 23:38:36.794034 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:36.793984 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9840bb1-3a88-4d36-8077-1facecc81158" path="/var/lib/kubelet/pods/a9840bb1-3a88-4d36-8077-1facecc81158/volumes" Nov 02 23:38:37.583719 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:38:37.583678 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:38:43.077571 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.077533 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8"] Nov 02 23:38:43.078120 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.078043 5114 cpu_manager.go:414] "containerMap: removing container" podUID="a9840bb1-3a88-4d36-8077-1facecc81158" containerName="istio-proxy" Nov 02 23:38:43.078120 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.078063 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9840bb1-3a88-4d36-8077-1facecc81158" containerName="istio-proxy" Nov 02 23:38:43.078120 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.078098 5114 cpu_manager.go:414] "containerMap: removing container" podUID="5693f409-a9e5-487a-885a-c254f8ef63cd" containerName="istio-proxy" Nov 02 23:38:43.078120 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.078103 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5693f409-a9e5-487a-885a-c254f8ef63cd" containerName="istio-proxy" Nov 02 23:38:43.078264 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.078190 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="5693f409-a9e5-487a-885a-c254f8ef63cd" containerName="istio-proxy" Nov 02 23:38:43.078264 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.078201 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9840bb1-3a88-4d36-8077-1facecc81158" containerName="istio-proxy" Nov 02 23:38:43.083013 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.082989 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:38:43.137782 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.137746 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-980b82\"/\"openshift-service-ca.crt\"" Nov 02 23:38:43.166069 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.166030 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-home\") pod \"auth-test-model-980b82-kserve-64ff84f86f-jsdq8\" (UID: \"27440af4-08bc-4f25-a76f-41a153b347e1\") " pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:38:43.166293 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.166078 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k976\" (UniqueName: \"kubernetes.io/projected/27440af4-08bc-4f25-a76f-41a153b347e1-kube-api-access-9k976\") pod \"auth-test-model-980b82-kserve-64ff84f86f-jsdq8\" (UID: \"27440af4-08bc-4f25-a76f-41a153b347e1\") " pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:38:43.166293 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.166206 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-model-cache\") pod \"auth-test-model-980b82-kserve-64ff84f86f-jsdq8\" (UID: \"27440af4-08bc-4f25-a76f-41a153b347e1\") " pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:38:43.166398 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.166378 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-dshm\") pod \"auth-test-model-980b82-kserve-64ff84f86f-jsdq8\" (UID: \"27440af4-08bc-4f25-a76f-41a153b347e1\") " pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:38:43.166545 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.166527 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-tmp-dir\") pod \"auth-test-model-980b82-kserve-64ff84f86f-jsdq8\" (UID: \"27440af4-08bc-4f25-a76f-41a153b347e1\") " pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:38:43.166621 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.166567 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27440af4-08bc-4f25-a76f-41a153b347e1-tls-certs\") pod \"auth-test-model-980b82-kserve-64ff84f86f-jsdq8\" (UID: \"27440af4-08bc-4f25-a76f-41a153b347e1\") " pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:38:43.187390 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.187345 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-980b82\"/\"default-dockercfg-vqkvp\"" Nov 02 23:38:43.187580 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.187433 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-980b82\"/\"auth-test-model-980b82-kserve-self-signed-certs\"" Nov 02 23:38:43.187670 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.187571 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-980b82\"/\"kube-root-ca.crt\"" Nov 02 23:38:43.189597 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.189573 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8"] Nov 02 23:38:43.267623 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.267564 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-model-cache\") pod \"auth-test-model-980b82-kserve-64ff84f86f-jsdq8\" (UID: \"27440af4-08bc-4f25-a76f-41a153b347e1\") " pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:38:43.267855 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.267706 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-dshm\") pod \"auth-test-model-980b82-kserve-64ff84f86f-jsdq8\" (UID: \"27440af4-08bc-4f25-a76f-41a153b347e1\") " pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:38:43.267855 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.267790 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-tmp-dir\") pod \"auth-test-model-980b82-kserve-64ff84f86f-jsdq8\" (UID: \"27440af4-08bc-4f25-a76f-41a153b347e1\") " pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:38:43.267855 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.267825 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27440af4-08bc-4f25-a76f-41a153b347e1-tls-certs\") pod \"auth-test-model-980b82-kserve-64ff84f86f-jsdq8\" (UID: \"27440af4-08bc-4f25-a76f-41a153b347e1\") " pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:38:43.268063 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.267890 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-home\") pod \"auth-test-model-980b82-kserve-64ff84f86f-jsdq8\" (UID: \"27440af4-08bc-4f25-a76f-41a153b347e1\") " pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:38:43.268063 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.267925 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k976\" (UniqueName: \"kubernetes.io/projected/27440af4-08bc-4f25-a76f-41a153b347e1-kube-api-access-9k976\") pod \"auth-test-model-980b82-kserve-64ff84f86f-jsdq8\" (UID: \"27440af4-08bc-4f25-a76f-41a153b347e1\") " pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:38:43.268558 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.268523 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-home\") pod \"auth-test-model-980b82-kserve-64ff84f86f-jsdq8\" (UID: \"27440af4-08bc-4f25-a76f-41a153b347e1\") " pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:38:43.269130 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.269109 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-tmp-dir\") pod \"auth-test-model-980b82-kserve-64ff84f86f-jsdq8\" (UID: \"27440af4-08bc-4f25-a76f-41a153b347e1\") " pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:38:43.269634 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.269612 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-model-cache\") pod \"auth-test-model-980b82-kserve-64ff84f86f-jsdq8\" (UID: \"27440af4-08bc-4f25-a76f-41a153b347e1\") " pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:38:43.271058 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.271036 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-dshm\") pod \"auth-test-model-980b82-kserve-64ff84f86f-jsdq8\" (UID: \"27440af4-08bc-4f25-a76f-41a153b347e1\") " pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:38:43.271432 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.271416 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27440af4-08bc-4f25-a76f-41a153b347e1-tls-certs\") pod \"auth-test-model-980b82-kserve-64ff84f86f-jsdq8\" (UID: \"27440af4-08bc-4f25-a76f-41a153b347e1\") " pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:38:43.329235 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.329148 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k976\" (UniqueName: \"kubernetes.io/projected/27440af4-08bc-4f25-a76f-41a153b347e1-kube-api-access-9k976\") pod \"auth-test-model-980b82-kserve-64ff84f86f-jsdq8\" (UID: \"27440af4-08bc-4f25-a76f-41a153b347e1\") " pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:38:43.395126 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.395092 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:38:43.780590 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:43.780563 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8"] Nov 02 23:38:43.782913 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:38:43.782882 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27440af4_08bc_4f25_a76f_41a153b347e1.slice/crio-3bbe98fbed07dab396a1f90736539a83b0f06cd4c0fdbd1176e87cc200b594d4 WatchSource:0}: Error finding container 3bbe98fbed07dab396a1f90736539a83b0f06cd4c0fdbd1176e87cc200b594d4: Status 404 returned error can't find the container with id 3bbe98fbed07dab396a1f90736539a83b0f06cd4c0fdbd1176e87cc200b594d4 Nov 02 23:38:44.510438 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:44.510397 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" event={"ID":"27440af4-08bc-4f25-a76f-41a153b347e1","Type":"ContainerStarted","Data":"650b74ddb968c1e152f50568c8d784b77f9f17371d2906ca5462b5b038d552ec"} Nov 02 23:38:44.510877 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:44.510451 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" event={"ID":"27440af4-08bc-4f25-a76f-41a153b347e1","Type":"ContainerStarted","Data":"3bbe98fbed07dab396a1f90736539a83b0f06cd4c0fdbd1176e87cc200b594d4"} Nov 02 23:38:44.628491 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:44.628422 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" podStartSLOduration=1.628400917 podStartE2EDuration="1.628400917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:38:44.62585836 +0000 UTC m=+2999.978891733" watchObservedRunningTime="2026-11-02 23:38:44.628400917 +0000 UTC m=+2999.981434296" Nov 02 23:38:45.395715 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:45.395678 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:38:45.490124 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:45.490094 5114 scope.go:122] "RemoveContainer" containerID="66e3ddaea0a119da1a5a92a8725eb35771f664bcf97d30a4f45fa7554786cb60" Nov 02 23:38:45.499427 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:45.499404 5114 scope.go:122] "RemoveContainer" containerID="27be659ab2f130365fc4df079795066a8ce7265745208e24126e8a25bd251932" Nov 02 23:38:53.395985 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:53.395937 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:38:53.410019 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:53.409991 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:38:53.579751 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:53.579720 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:38:57.288087 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:57.288041 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx"] Nov 02 23:38:57.288747 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:38:57.288286 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" podUID="0003757b-3035-4a01-9aac-0f9f10c0c845" containerName="istio-proxy" containerID="cri-o://bf4c7ef11e8fcd50c4cee8a0e517172355b695f75fe4844932787552a8e937d8" gracePeriod=30 Nov 02 23:39:02.556537 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.556435 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:39:02.619981 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.619946 5114 generic.go:356] "Generic (PLEG): container finished" podID="0003757b-3035-4a01-9aac-0f9f10c0c845" containerID="bf4c7ef11e8fcd50c4cee8a0e517172355b695f75fe4844932787552a8e937d8" exitCode=0 Nov 02 23:39:02.620162 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.620004 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" Nov 02 23:39:02.620162 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.620031 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" event={"ID":"0003757b-3035-4a01-9aac-0f9f10c0c845","Type":"ContainerDied","Data":"bf4c7ef11e8fcd50c4cee8a0e517172355b695f75fe4844932787552a8e937d8"} Nov 02 23:39:02.620162 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.620093 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx" event={"ID":"0003757b-3035-4a01-9aac-0f9f10c0c845","Type":"ContainerDied","Data":"43ab9d8c11c235f8cb9012e95a5b743b33eb253efbd17346ef426b8aaade9085"} Nov 02 23:39:02.620162 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.620121 5114 scope.go:122] "RemoveContainer" containerID="bf4c7ef11e8fcd50c4cee8a0e517172355b695f75fe4844932787552a8e937d8" Nov 02 23:39:02.630115 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.630091 5114 scope.go:122] "RemoveContainer" containerID="bf4c7ef11e8fcd50c4cee8a0e517172355b695f75fe4844932787552a8e937d8" Nov 02 23:39:02.630424 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:39:02.630406 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf4c7ef11e8fcd50c4cee8a0e517172355b695f75fe4844932787552a8e937d8\": container with ID starting with bf4c7ef11e8fcd50c4cee8a0e517172355b695f75fe4844932787552a8e937d8 not found: ID does not exist" containerID="bf4c7ef11e8fcd50c4cee8a0e517172355b695f75fe4844932787552a8e937d8" Nov 02 23:39:02.630477 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.630439 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf4c7ef11e8fcd50c4cee8a0e517172355b695f75fe4844932787552a8e937d8"} err="failed to get container status \"bf4c7ef11e8fcd50c4cee8a0e517172355b695f75fe4844932787552a8e937d8\": rpc error: code = NotFound desc = could not find container \"bf4c7ef11e8fcd50c4cee8a0e517172355b695f75fe4844932787552a8e937d8\": container with ID starting with bf4c7ef11e8fcd50c4cee8a0e517172355b695f75fe4844932787552a8e937d8 not found: ID does not exist" Nov 02 23:39:02.707534 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.707494 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-credential-socket\") pod \"0003757b-3035-4a01-9aac-0f9f10c0c845\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " Nov 02 23:39:02.707723 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.707596 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-podinfo\") pod \"0003757b-3035-4a01-9aac-0f9f10c0c845\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " Nov 02 23:39:02.707723 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.707630 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-data\") pod \"0003757b-3035-4a01-9aac-0f9f10c0c845\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " Nov 02 23:39:02.707723 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.707689 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0003757b-3035-4a01-9aac-0f9f10c0c845-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0003757b-3035-4a01-9aac-0f9f10c0c845-istiod-ca-cert\") pod \"0003757b-3035-4a01-9aac-0f9f10c0c845\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " Nov 02 23:39:02.707902 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.707734 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0003757b-3035-4a01-9aac-0f9f10c0c845-kube-api-access-hqtdc\" (UniqueName: \"kubernetes.io/projected/0003757b-3035-4a01-9aac-0f9f10c0c845-kube-api-access-hqtdc\") pod \"0003757b-3035-4a01-9aac-0f9f10c0c845\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " Nov 02 23:39:02.707902 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.707841 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-envoy\") pod \"0003757b-3035-4a01-9aac-0f9f10c0c845\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " Nov 02 23:39:02.707902 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.707874 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-workload-socket\") pod \"0003757b-3035-4a01-9aac-0f9f10c0c845\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " Nov 02 23:39:02.708064 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.707946 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-workload-certs\") pod \"0003757b-3035-4a01-9aac-0f9f10c0c845\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " Nov 02 23:39:02.708064 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.707992 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-token\" (UniqueName: \"kubernetes.io/projected/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-token\") pod \"0003757b-3035-4a01-9aac-0f9f10c0c845\" (UID: \"0003757b-3035-4a01-9aac-0f9f10c0c845\") " Nov 02 23:39:02.708064 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.708007 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-data" pod "0003757b-3035-4a01-9aac-0f9f10c0c845" (UID: "0003757b-3035-4a01-9aac-0f9f10c0c845"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:39:02.708223 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.708121 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0003757b-3035-4a01-9aac-0f9f10c0c845-istiod-ca-cert" pod "0003757b-3035-4a01-9aac-0f9f10c0c845" (UID: "0003757b-3035-4a01-9aac-0f9f10c0c845"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:39:02.708279 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.708246 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-workload-certs" pod "0003757b-3035-4a01-9aac-0f9f10c0c845" (UID: "0003757b-3035-4a01-9aac-0f9f10c0c845"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:39:02.708448 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.707817 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-credential-socket" pod "0003757b-3035-4a01-9aac-0f9f10c0c845" (UID: "0003757b-3035-4a01-9aac-0f9f10c0c845"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:39:02.708660 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.708615 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-workload-socket" pod "0003757b-3035-4a01-9aac-0f9f10c0c845" (UID: "0003757b-3035-4a01-9aac-0f9f10c0c845"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:39:02.708807 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.708778 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:02.708807 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.708804 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:02.708942 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.708819 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0003757b-3035-4a01-9aac-0f9f10c0c845-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:02.708942 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.708833 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:02.708942 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.708847 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:02.710342 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.710312 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-podinfo" pod "0003757b-3035-4a01-9aac-0f9f10c0c845" (UID: "0003757b-3035-4a01-9aac-0f9f10c0c845"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:39:02.710659 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.710633 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0003757b-3035-4a01-9aac-0f9f10c0c845-kube-api-access-hqtdc" pod "0003757b-3035-4a01-9aac-0f9f10c0c845" (UID: "0003757b-3035-4a01-9aac-0f9f10c0c845"). InnerVolumeSpecName "kube-api-access-hqtdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:39:02.710884 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.710864 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-envoy" pod "0003757b-3035-4a01-9aac-0f9f10c0c845" (UID: "0003757b-3035-4a01-9aac-0f9f10c0c845"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:39:02.711054 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.711035 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-token" pod "0003757b-3035-4a01-9aac-0f9f10c0c845" (UID: "0003757b-3035-4a01-9aac-0f9f10c0c845"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:39:02.810168 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.810065 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:02.810168 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.810102 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:02.810168 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.810117 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqtdc\" (UniqueName: \"kubernetes.io/projected/0003757b-3035-4a01-9aac-0f9f10c0c845-kube-api-access-hqtdc\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:02.810168 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:02.810132 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0003757b-3035-4a01-9aac-0f9f10c0c845-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:03.036559 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:03.036517 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx"] Nov 02 23:39:03.040532 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:03.040499 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-pcvpx"] Nov 02 23:39:04.793271 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:04.793230 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0003757b-3035-4a01-9aac-0f9f10c0c845" path="/var/lib/kubelet/pods/0003757b-3035-4a01-9aac-0f9f10c0c845/volumes" Nov 02 23:39:06.285470 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:06.285433 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws"] Nov 02 23:39:07.583921 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:39:07.583884 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:39:21.310066 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.309989 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" podUID="6d314a4e-a7f3-41b6-b2e5-671bc57d9c62" containerName="main" containerID="cri-o://5505337994c33da7ba1434295c6a1f285d68d351978390ec82445c3737865332" gracePeriod=45 Nov 02 23:39:21.576628 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.576554 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:39:21.726084 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.726042 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-tmp-dir\") pod \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\" (UID: \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\") " Nov 02 23:39:21.726292 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.726106 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-dshm\") pod \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\" (UID: \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\") " Nov 02 23:39:21.726292 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.726172 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-tls-certs\") pod \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\" (UID: \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\") " Nov 02 23:39:21.726292 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.726220 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-model-cache\") pod \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\" (UID: \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\") " Nov 02 23:39:21.726292 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.726272 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-home\" (UniqueName: \"kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-home\") pod \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\" (UID: \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\") " Nov 02 23:39:21.726536 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.726321 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-kube-api-access-v7jm6\" (UniqueName: \"kubernetes.io/projected/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-kube-api-access-v7jm6\") pod \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\" (UID: \"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62\") " Nov 02 23:39:21.726536 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.726376 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-tmp-dir" pod "6d314a4e-a7f3-41b6-b2e5-671bc57d9c62" (UID: "6d314a4e-a7f3-41b6-b2e5-671bc57d9c62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:39:21.726647 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.726572 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-model-cache" pod "6d314a4e-a7f3-41b6-b2e5-671bc57d9c62" (UID: "6d314a4e-a7f3-41b6-b2e5-671bc57d9c62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:39:21.726805 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.726783 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:21.726869 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.726812 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:21.726947 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.726924 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-home" pod "6d314a4e-a7f3-41b6-b2e5-671bc57d9c62" (UID: "6d314a4e-a7f3-41b6-b2e5-671bc57d9c62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:39:21.729301 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.729266 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-tls-certs" pod "6d314a4e-a7f3-41b6-b2e5-671bc57d9c62" (UID: "6d314a4e-a7f3-41b6-b2e5-671bc57d9c62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:39:21.729455 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.729360 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-dshm" pod "6d314a4e-a7f3-41b6-b2e5-671bc57d9c62" (UID: "6d314a4e-a7f3-41b6-b2e5-671bc57d9c62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:39:21.729557 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.729527 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-kube-api-access-v7jm6" pod "6d314a4e-a7f3-41b6-b2e5-671bc57d9c62" (UID: "6d314a4e-a7f3-41b6-b2e5-671bc57d9c62"). InnerVolumeSpecName "kube-api-access-v7jm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:39:21.729666 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.729643 5114 generic.go:356] "Generic (PLEG): container finished" podID="6d314a4e-a7f3-41b6-b2e5-671bc57d9c62" containerID="5505337994c33da7ba1434295c6a1f285d68d351978390ec82445c3737865332" exitCode=0 Nov 02 23:39:21.729756 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.729706 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" Nov 02 23:39:21.729756 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.729723 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" event={"ID":"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62","Type":"ContainerDied","Data":"5505337994c33da7ba1434295c6a1f285d68d351978390ec82445c3737865332"} Nov 02 23:39:21.729877 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.729767 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws" event={"ID":"6d314a4e-a7f3-41b6-b2e5-671bc57d9c62","Type":"ContainerDied","Data":"5970272b15ebb2cda7a233822e1549dede83069226acd6190e5f37738237f1a9"} Nov 02 23:39:21.729877 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.729789 5114 scope.go:122] "RemoveContainer" containerID="5505337994c33da7ba1434295c6a1f285d68d351978390ec82445c3737865332" Nov 02 23:39:21.752272 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.752246 5114 scope.go:122] "RemoveContainer" containerID="5505337994c33da7ba1434295c6a1f285d68d351978390ec82445c3737865332" Nov 02 23:39:21.752669 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:39:21.752643 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5505337994c33da7ba1434295c6a1f285d68d351978390ec82445c3737865332\": container with ID starting with 5505337994c33da7ba1434295c6a1f285d68d351978390ec82445c3737865332 not found: ID does not exist" containerID="5505337994c33da7ba1434295c6a1f285d68d351978390ec82445c3737865332" Nov 02 23:39:21.752771 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.752684 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5505337994c33da7ba1434295c6a1f285d68d351978390ec82445c3737865332"} err="failed to get container status \"5505337994c33da7ba1434295c6a1f285d68d351978390ec82445c3737865332\": rpc error: code = NotFound desc = could not find container \"5505337994c33da7ba1434295c6a1f285d68d351978390ec82445c3737865332\": container with ID starting with 5505337994c33da7ba1434295c6a1f285d68d351978390ec82445c3737865332 not found: ID does not exist" Nov 02 23:39:21.828601 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.828465 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:21.828601 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.828540 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:21.828601 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.828549 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:21.828601 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.828558 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7jm6\" (UniqueName: \"kubernetes.io/projected/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62-kube-api-access-v7jm6\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:21.892652 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.892614 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws"] Nov 02 23:39:21.896576 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:21.896545 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-94e3ae/praxis-route-94e3ae-kserve-659d6895c5-4zjws"] Nov 02 23:39:22.794661 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:22.794619 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d314a4e-a7f3-41b6-b2e5-671bc57d9c62" path="/var/lib/kubelet/pods/6d314a4e-a7f3-41b6-b2e5-671bc57d9c62/volumes" Nov 02 23:39:35.711288 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.711248 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz"] Nov 02 23:39:35.711924 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.711900 5114 cpu_manager.go:414] "containerMap: removing container" podUID="6d314a4e-a7f3-41b6-b2e5-671bc57d9c62" containerName="main" Nov 02 23:39:35.712036 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.711928 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d314a4e-a7f3-41b6-b2e5-671bc57d9c62" containerName="main" Nov 02 23:39:35.712036 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.711950 5114 cpu_manager.go:414] "containerMap: removing container" podUID="0003757b-3035-4a01-9aac-0f9f10c0c845" containerName="istio-proxy" Nov 02 23:39:35.712036 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.711958 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0003757b-3035-4a01-9aac-0f9f10c0c845" containerName="istio-proxy" Nov 02 23:39:35.712187 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.712076 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="0003757b-3035-4a01-9aac-0f9f10c0c845" containerName="istio-proxy" Nov 02 23:39:35.712187 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.712099 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d314a4e-a7f3-41b6-b2e5-671bc57d9c62" containerName="main" Nov 02 23:39:35.716743 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.716717 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:39:35.771986 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.771955 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1f0009\"/\"kube-root-ca.crt\"" Nov 02 23:39:35.776631 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.776598 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-home\") pod \"auth-test-model-1f0009-kserve-7686cff4f7-xd4xz\" (UID: \"737c5cf0-065a-4223-97be-3176aebab7d6\") " pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:39:35.776631 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.776644 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-model-cache\") pod \"auth-test-model-1f0009-kserve-7686cff4f7-xd4xz\" (UID: \"737c5cf0-065a-4223-97be-3176aebab7d6\") " pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:39:35.776857 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.776747 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-dshm\") pod \"auth-test-model-1f0009-kserve-7686cff4f7-xd4xz\" (UID: \"737c5cf0-065a-4223-97be-3176aebab7d6\") " pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:39:35.776857 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.776845 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-tmp-dir\") pod \"auth-test-model-1f0009-kserve-7686cff4f7-xd4xz\" (UID: \"737c5cf0-065a-4223-97be-3176aebab7d6\") " pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:39:35.776938 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.776877 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/737c5cf0-065a-4223-97be-3176aebab7d6-tls-certs\") pod \"auth-test-model-1f0009-kserve-7686cff4f7-xd4xz\" (UID: \"737c5cf0-065a-4223-97be-3176aebab7d6\") " pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:39:35.777028 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.777013 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6nf9\" (UniqueName: \"kubernetes.io/projected/737c5cf0-065a-4223-97be-3176aebab7d6-kube-api-access-w6nf9\") pod \"auth-test-model-1f0009-kserve-7686cff4f7-xd4xz\" (UID: \"737c5cf0-065a-4223-97be-3176aebab7d6\") " pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:39:35.821620 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.821583 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1f0009\"/\"auth-test-model-1f0009-kserve-self-signed-certs\"" Nov 02 23:39:35.821620 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.821583 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1f0009\"/\"openshift-service-ca.crt\"" Nov 02 23:39:35.821844 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.821585 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1f0009\"/\"default-dockercfg-d9tml\"" Nov 02 23:39:35.823765 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.823745 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz"] Nov 02 23:39:35.877940 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.877902 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-home\") pod \"auth-test-model-1f0009-kserve-7686cff4f7-xd4xz\" (UID: \"737c5cf0-065a-4223-97be-3176aebab7d6\") " pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:39:35.877940 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.877957 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-model-cache\") pod \"auth-test-model-1f0009-kserve-7686cff4f7-xd4xz\" (UID: \"737c5cf0-065a-4223-97be-3176aebab7d6\") " pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:39:35.878203 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.878001 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-dshm\") pod \"auth-test-model-1f0009-kserve-7686cff4f7-xd4xz\" (UID: \"737c5cf0-065a-4223-97be-3176aebab7d6\") " pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:39:35.878203 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.878059 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-tmp-dir\") pod \"auth-test-model-1f0009-kserve-7686cff4f7-xd4xz\" (UID: \"737c5cf0-065a-4223-97be-3176aebab7d6\") " pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:39:35.878203 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.878096 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/737c5cf0-065a-4223-97be-3176aebab7d6-tls-certs\") pod \"auth-test-model-1f0009-kserve-7686cff4f7-xd4xz\" (UID: \"737c5cf0-065a-4223-97be-3176aebab7d6\") " pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:39:35.878344 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.878207 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6nf9\" (UniqueName: \"kubernetes.io/projected/737c5cf0-065a-4223-97be-3176aebab7d6-kube-api-access-w6nf9\") pod \"auth-test-model-1f0009-kserve-7686cff4f7-xd4xz\" (UID: \"737c5cf0-065a-4223-97be-3176aebab7d6\") " pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:39:35.878991 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.878907 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-home\") pod \"auth-test-model-1f0009-kserve-7686cff4f7-xd4xz\" (UID: \"737c5cf0-065a-4223-97be-3176aebab7d6\") " pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:39:35.879867 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.879794 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-tmp-dir\") pod \"auth-test-model-1f0009-kserve-7686cff4f7-xd4xz\" (UID: \"737c5cf0-065a-4223-97be-3176aebab7d6\") " pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:39:35.880034 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.879919 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-model-cache\") pod \"auth-test-model-1f0009-kserve-7686cff4f7-xd4xz\" (UID: \"737c5cf0-065a-4223-97be-3176aebab7d6\") " pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:39:35.882849 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.882752 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-dshm\") pod \"auth-test-model-1f0009-kserve-7686cff4f7-xd4xz\" (UID: \"737c5cf0-065a-4223-97be-3176aebab7d6\") " pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:39:35.884165 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.884143 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/737c5cf0-065a-4223-97be-3176aebab7d6-tls-certs\") pod \"auth-test-model-1f0009-kserve-7686cff4f7-xd4xz\" (UID: \"737c5cf0-065a-4223-97be-3176aebab7d6\") " pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:39:35.939577 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:35.939536 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6nf9\" (UniqueName: \"kubernetes.io/projected/737c5cf0-065a-4223-97be-3176aebab7d6-kube-api-access-w6nf9\") pod \"auth-test-model-1f0009-kserve-7686cff4f7-xd4xz\" (UID: \"737c5cf0-065a-4223-97be-3176aebab7d6\") " pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:39:36.029594 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:36.029457 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:39:36.410634 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:36.410574 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz"] Nov 02 23:39:36.413135 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:39:36.413091 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod737c5cf0_065a_4223_97be_3176aebab7d6.slice/crio-323adfb9d7e2bc57ad3f18f5d1923f28a9dfca4bb65961958fbd5643b2d45660 WatchSource:0}: Error finding container 323adfb9d7e2bc57ad3f18f5d1923f28a9dfca4bb65961958fbd5643b2d45660: Status 404 returned error can't find the container with id 323adfb9d7e2bc57ad3f18f5d1923f28a9dfca4bb65961958fbd5643b2d45660 Nov 02 23:39:36.815086 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:36.815038 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" event={"ID":"737c5cf0-065a-4223-97be-3176aebab7d6","Type":"ContainerStarted","Data":"c3989486acdf4af45a0d698e0e268c1ab4aa0e8d8df4fd08f3134a7a8594fa05"} Nov 02 23:39:36.815560 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:36.815103 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" event={"ID":"737c5cf0-065a-4223-97be-3176aebab7d6","Type":"ContainerStarted","Data":"323adfb9d7e2bc57ad3f18f5d1923f28a9dfca4bb65961958fbd5643b2d45660"} Nov 02 23:39:36.932215 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:36.932152 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" podStartSLOduration=1.9321356280000002 podStartE2EDuration="1.932135628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:39:36.931270578 +0000 UTC m=+3052.284303954" watchObservedRunningTime="2026-11-02 23:39:36.932135628 +0000 UTC m=+3052.285169060" Nov 02 23:39:37.029853 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:37.029815 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:39:37.590590 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:39:37.590550 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:39:42.993770 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:42.993731 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86a28697-7wkdc"] Nov 02 23:39:42.997650 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:42.997623 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a28697-7wkdc" Nov 02 23:39:43.066122 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:43.066073 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kptp7\" (UniqueName: \"kubernetes.io/projected/e7d12b96-e660-4927-9424-b5ce36b78ac5-kube-api-access-kptp7\") pod \"maas-api-revoke-sub-86a28697-7wkdc\" (UID: \"e7d12b96-e660-4927-9424-b5ce36b78ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a28697-7wkdc" Nov 02 23:39:43.066389 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:43.066189 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7d12b96-e660-4927-9424-b5ce36b78ac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86a28697-7wkdc\" (UID: \"e7d12b96-e660-4927-9424-b5ce36b78ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a28697-7wkdc" Nov 02 23:39:43.066389 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:43.066264 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7d12b96-e660-4927-9424-b5ce36b78ac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86a28697-7wkdc\" (UID: \"e7d12b96-e660-4927-9424-b5ce36b78ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a28697-7wkdc" Nov 02 23:39:43.095257 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:43.095224 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:39:43.101378 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:43.101349 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:39:43.103523 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:43.103492 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86a28697-7wkdc"] Nov 02 23:39:43.167548 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:43.167468 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7d12b96-e660-4927-9424-b5ce36b78ac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86a28697-7wkdc\" (UID: \"e7d12b96-e660-4927-9424-b5ce36b78ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a28697-7wkdc" Nov 02 23:39:43.167761 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:43.167740 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kptp7\" (UniqueName: \"kubernetes.io/projected/e7d12b96-e660-4927-9424-b5ce36b78ac5-kube-api-access-kptp7\") pod \"maas-api-revoke-sub-86a28697-7wkdc\" (UID: \"e7d12b96-e660-4927-9424-b5ce36b78ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a28697-7wkdc" Nov 02 23:39:43.167819 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:43.167790 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7d12b96-e660-4927-9424-b5ce36b78ac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86a28697-7wkdc\" (UID: \"e7d12b96-e660-4927-9424-b5ce36b78ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a28697-7wkdc" Nov 02 23:39:43.168509 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:43.168468 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7d12b96-e660-4927-9424-b5ce36b78ac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86a28697-7wkdc\" (UID: \"e7d12b96-e660-4927-9424-b5ce36b78ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a28697-7wkdc" Nov 02 23:39:43.170337 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:43.170315 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7d12b96-e660-4927-9424-b5ce36b78ac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86a28697-7wkdc\" (UID: \"e7d12b96-e660-4927-9424-b5ce36b78ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a28697-7wkdc" Nov 02 23:39:43.229441 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:43.229404 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kptp7\" (UniqueName: \"kubernetes.io/projected/e7d12b96-e660-4927-9424-b5ce36b78ac5-kube-api-access-kptp7\") pod \"maas-api-revoke-sub-86a28697-7wkdc\" (UID: \"e7d12b96-e660-4927-9424-b5ce36b78ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a28697-7wkdc" Nov 02 23:39:43.318774 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:43.318732 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a28697-7wkdc" Nov 02 23:39:43.709664 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:43.709631 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86a28697-7wkdc"] Nov 02 23:39:43.711985 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:39:43.711953 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d12b96_e660_4927_9424_b5ce36b78ac5.slice/crio-99a3bb0048019f9eb1c18005f1116644a9628f77fd3206699f102784b35a5a3b WatchSource:0}: Error finding container 99a3bb0048019f9eb1c18005f1116644a9628f77fd3206699f102784b35a5a3b: Status 404 returned error can't find the container with id 99a3bb0048019f9eb1c18005f1116644a9628f77fd3206699f102784b35a5a3b Nov 02 23:39:43.854928 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:43.854891 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a28697-7wkdc" event={"ID":"e7d12b96-e660-4927-9424-b5ce36b78ac5","Type":"ContainerStarted","Data":"99a3bb0048019f9eb1c18005f1116644a9628f77fd3206699f102784b35a5a3b"} Nov 02 23:39:44.861623 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:44.861584 5114 generic.go:356] "Generic (PLEG): container finished" podID="e7d12b96-e660-4927-9424-b5ce36b78ac5" containerID="f7319cffef8ab05c787c4af6a33e24af546d50f89b75ecae45affef9985e0c2c" exitCode=0 Nov 02 23:39:44.862013 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:44.861632 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a28697-7wkdc" event={"ID":"e7d12b96-e660-4927-9424-b5ce36b78ac5","Type":"ContainerDied","Data":"f7319cffef8ab05c787c4af6a33e24af546d50f89b75ecae45affef9985e0c2c"} Nov 02 23:39:45.538499 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:45.538460 5114 scope.go:122] "RemoveContainer" containerID="9b3431e05c7373352f50390c7af249aed18557608b770a7b6c35c638f7a6e101" Nov 02 23:39:46.019255 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:46.019219 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a28697-7wkdc" Nov 02 23:39:46.029840 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:46.029802 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:39:46.042910 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:46.042876 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:39:46.106425 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:46.106383 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7d12b96-e660-4927-9424-b5ce36b78ac5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7d12b96-e660-4927-9424-b5ce36b78ac5-maas-api-xks-ca\") pod \"e7d12b96-e660-4927-9424-b5ce36b78ac5\" (UID: \"e7d12b96-e660-4927-9424-b5ce36b78ac5\") " Nov 02 23:39:46.106646 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:46.106541 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7d12b96-e660-4927-9424-b5ce36b78ac5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7d12b96-e660-4927-9424-b5ce36b78ac5-maas-api-openshift-ca\") pod \"e7d12b96-e660-4927-9424-b5ce36b78ac5\" (UID: \"e7d12b96-e660-4927-9424-b5ce36b78ac5\") " Nov 02 23:39:46.106646 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:46.106613 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7d12b96-e660-4927-9424-b5ce36b78ac5-kube-api-access-kptp7\" (UniqueName: \"kubernetes.io/projected/e7d12b96-e660-4927-9424-b5ce36b78ac5-kube-api-access-kptp7\") pod \"e7d12b96-e660-4927-9424-b5ce36b78ac5\" (UID: \"e7d12b96-e660-4927-9424-b5ce36b78ac5\") " Nov 02 23:39:46.106930 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:46.106888 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7d12b96-e660-4927-9424-b5ce36b78ac5-maas-api-openshift-ca" pod "e7d12b96-e660-4927-9424-b5ce36b78ac5" (UID: "e7d12b96-e660-4927-9424-b5ce36b78ac5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:39:46.107128 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:46.107105 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7d12b96-e660-4927-9424-b5ce36b78ac5-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:46.109105 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:46.109071 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d12b96-e660-4927-9424-b5ce36b78ac5-kube-api-access-kptp7" pod "e7d12b96-e660-4927-9424-b5ce36b78ac5" (UID: "e7d12b96-e660-4927-9424-b5ce36b78ac5"). InnerVolumeSpecName "kube-api-access-kptp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:39:46.109257 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:46.109160 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7d12b96-e660-4927-9424-b5ce36b78ac5-maas-api-xks-ca" pod "e7d12b96-e660-4927-9424-b5ce36b78ac5" (UID: "e7d12b96-e660-4927-9424-b5ce36b78ac5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:39:46.208058 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:46.207965 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7d12b96-e660-4927-9424-b5ce36b78ac5-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:46.208058 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:46.208007 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kptp7\" (UniqueName: \"kubernetes.io/projected/e7d12b96-e660-4927-9424-b5ce36b78ac5-kube-api-access-kptp7\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:39:46.872916 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:46.872881 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a28697-7wkdc" Nov 02 23:39:46.873126 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:46.872921 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a28697-7wkdc" event={"ID":"e7d12b96-e660-4927-9424-b5ce36b78ac5","Type":"ContainerDied","Data":"99a3bb0048019f9eb1c18005f1116644a9628f77fd3206699f102784b35a5a3b"} Nov 02 23:39:46.873126 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:46.872987 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99a3bb0048019f9eb1c18005f1116644a9628f77fd3206699f102784b35a5a3b" Nov 02 23:39:46.885283 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:39:46.885251 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:40:07.589199 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:40:07.589147 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:40:27.973456 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:27.973414 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb"] Nov 02 23:40:27.973926 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:27.973644 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" podUID="7bd95eaa-10d3-4928-bf81-adcd1981f259" containerName="istio-proxy" containerID="cri-o://85aee179346b724fab77c7db797099ff9ea2790a132a7ac53bee3cd267ec5200" gracePeriod=30 Nov 02 23:40:31.156154 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:31.156114 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv"] Nov 02 23:40:31.156712 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:31.156385 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" podUID="8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5" containerName="maas-api" containerID="cri-o://e30dcd68a7c825de8809dad376c20016408b4ad1c5f9116e26f04b5f5717d75e" gracePeriod=30 Nov 02 23:40:31.469215 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:31.469175 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" Nov 02 23:40:31.541289 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:31.541246 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-kube-api-access-rkgs7\" (UniqueName: \"kubernetes.io/projected/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-kube-api-access-rkgs7\") pod \"8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5\" (UID: \"8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5\") " Nov 02 23:40:31.541536 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:31.541436 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-metrics-certs\") pod \"8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5\" (UID: \"8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5\") " Nov 02 23:40:31.541647 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:31.541629 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-openshift-service-ca\") pod \"8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5\" (UID: \"8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5\") " Nov 02 23:40:31.541723 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:31.541688 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-maas-api-tls\") pod \"8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5\" (UID: \"8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5\") " Nov 02 23:40:31.542238 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:31.542193 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-openshift-service-ca" pod "8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5" (UID: "8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:40:31.545076 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:31.545024 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-maas-api-tls" pod "8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5" (UID: "8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:40:31.545380 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:31.545345 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-metrics-certs" pod "8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5" (UID: "8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:40:31.545901 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:31.545864 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-kube-api-access-rkgs7" pod "8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5" (UID: "8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5"). InnerVolumeSpecName "kube-api-access-rkgs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:40:31.643117 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:31.643075 5114 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-openshift-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:31.643117 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:31.643112 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-maas-api-tls\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:31.643117 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:31.643123 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkgs7\" (UniqueName: \"kubernetes.io/projected/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-kube-api-access-rkgs7\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:31.643378 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:31.643132 5114 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5-metrics-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:32.129687 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:32.129643 5114 generic.go:356] "Generic (PLEG): container finished" podID="8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5" containerID="e30dcd68a7c825de8809dad376c20016408b4ad1c5f9116e26f04b5f5717d75e" exitCode=0 Nov 02 23:40:32.129875 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:32.129695 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" Nov 02 23:40:32.129875 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:32.129732 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" event={"ID":"8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5","Type":"ContainerDied","Data":"e30dcd68a7c825de8809dad376c20016408b4ad1c5f9116e26f04b5f5717d75e"} Nov 02 23:40:32.129875 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:32.129797 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv" event={"ID":"8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5","Type":"ContainerDied","Data":"911fdaad888322d56e136737e8623b01a1308782700ffae4f02cfc2777011fce"} Nov 02 23:40:32.129875 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:32.129833 5114 scope.go:122] "RemoveContainer" containerID="e30dcd68a7c825de8809dad376c20016408b4ad1c5f9116e26f04b5f5717d75e" Nov 02 23:40:32.145410 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:32.145381 5114 scope.go:122] "RemoveContainer" containerID="e30dcd68a7c825de8809dad376c20016408b4ad1c5f9116e26f04b5f5717d75e" Nov 02 23:40:32.145887 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:40:32.145862 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e30dcd68a7c825de8809dad376c20016408b4ad1c5f9116e26f04b5f5717d75e\": container with ID starting with e30dcd68a7c825de8809dad376c20016408b4ad1c5f9116e26f04b5f5717d75e not found: ID does not exist" containerID="e30dcd68a7c825de8809dad376c20016408b4ad1c5f9116e26f04b5f5717d75e" Nov 02 23:40:32.145982 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:32.145903 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e30dcd68a7c825de8809dad376c20016408b4ad1c5f9116e26f04b5f5717d75e"} err="failed to get container status \"e30dcd68a7c825de8809dad376c20016408b4ad1c5f9116e26f04b5f5717d75e\": rpc error: code = NotFound desc = could not find container \"e30dcd68a7c825de8809dad376c20016408b4ad1c5f9116e26f04b5f5717d75e\": container with ID starting with e30dcd68a7c825de8809dad376c20016408b4ad1c5f9116e26f04b5f5717d75e not found: ID does not exist" Nov 02 23:40:32.297789 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:32.297745 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv"] Nov 02 23:40:32.302864 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:32.302838 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-94e3ae-5fbdf4d465-45ngv"] Nov 02 23:40:32.793559 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:32.793521 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5" path="/var/lib/kubelet/pods/8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5/volumes" Nov 02 23:40:33.137474 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.137429 5114 generic.go:356] "Generic (PLEG): container finished" podID="7bd95eaa-10d3-4928-bf81-adcd1981f259" containerID="85aee179346b724fab77c7db797099ff9ea2790a132a7ac53bee3cd267ec5200" exitCode=0 Nov 02 23:40:33.137691 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.137513 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" event={"ID":"7bd95eaa-10d3-4928-bf81-adcd1981f259","Type":"ContainerDied","Data":"85aee179346b724fab77c7db797099ff9ea2790a132a7ac53bee3cd267ec5200"} Nov 02 23:40:33.241660 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.241630 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:40:33.363908 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.363798 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-token\" (UniqueName: \"kubernetes.io/projected/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-token\") pod \"7bd95eaa-10d3-4928-bf81-adcd1981f259\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " Nov 02 23:40:33.363908 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.363853 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd95eaa-10d3-4928-bf81-adcd1981f259-kube-api-access-v6ztx\" (UniqueName: \"kubernetes.io/projected/7bd95eaa-10d3-4928-bf81-adcd1981f259-kube-api-access-v6ztx\") pod \"7bd95eaa-10d3-4928-bf81-adcd1981f259\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " Nov 02 23:40:33.365072 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.363946 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-workload-certs\") pod \"7bd95eaa-10d3-4928-bf81-adcd1981f259\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " Nov 02 23:40:33.365072 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.363997 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-podinfo\") pod \"7bd95eaa-10d3-4928-bf81-adcd1981f259\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " Nov 02 23:40:33.365072 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.364029 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-credential-socket\") pod \"7bd95eaa-10d3-4928-bf81-adcd1981f259\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " Nov 02 23:40:33.365072 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.364080 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bd95eaa-10d3-4928-bf81-adcd1981f259-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd95eaa-10d3-4928-bf81-adcd1981f259-istiod-ca-cert\") pod \"7bd95eaa-10d3-4928-bf81-adcd1981f259\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " Nov 02 23:40:33.365072 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.364115 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-envoy\") pod \"7bd95eaa-10d3-4928-bf81-adcd1981f259\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " Nov 02 23:40:33.365072 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.364147 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-data\") pod \"7bd95eaa-10d3-4928-bf81-adcd1981f259\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " Nov 02 23:40:33.365072 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.364201 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-workload-socket\") pod \"7bd95eaa-10d3-4928-bf81-adcd1981f259\" (UID: \"7bd95eaa-10d3-4928-bf81-adcd1981f259\") " Nov 02 23:40:33.365072 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.364917 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-workload-socket" pod "7bd95eaa-10d3-4928-bf81-adcd1981f259" (UID: "7bd95eaa-10d3-4928-bf81-adcd1981f259"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:40:33.365638 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.365420 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-workload-certs" pod "7bd95eaa-10d3-4928-bf81-adcd1981f259" (UID: "7bd95eaa-10d3-4928-bf81-adcd1981f259"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:40:33.365985 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.365953 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-data" pod "7bd95eaa-10d3-4928-bf81-adcd1981f259" (UID: "7bd95eaa-10d3-4928-bf81-adcd1981f259"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:40:33.366543 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.366330 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-credential-socket" pod "7bd95eaa-10d3-4928-bf81-adcd1981f259" (UID: "7bd95eaa-10d3-4928-bf81-adcd1981f259"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:40:33.366543 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.366412 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd95eaa-10d3-4928-bf81-adcd1981f259-istiod-ca-cert" pod "7bd95eaa-10d3-4928-bf81-adcd1981f259" (UID: "7bd95eaa-10d3-4928-bf81-adcd1981f259"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:40:33.369152 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.369127 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-podinfo" pod "7bd95eaa-10d3-4928-bf81-adcd1981f259" (UID: "7bd95eaa-10d3-4928-bf81-adcd1981f259"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:40:33.369305 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.369276 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd95eaa-10d3-4928-bf81-adcd1981f259-kube-api-access-v6ztx" pod "7bd95eaa-10d3-4928-bf81-adcd1981f259" (UID: "7bd95eaa-10d3-4928-bf81-adcd1981f259"). InnerVolumeSpecName "kube-api-access-v6ztx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:40:33.370110 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.370086 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-envoy" pod "7bd95eaa-10d3-4928-bf81-adcd1981f259" (UID: "7bd95eaa-10d3-4928-bf81-adcd1981f259"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:40:33.370624 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.370595 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-token" pod "7bd95eaa-10d3-4928-bf81-adcd1981f259" (UID: "7bd95eaa-10d3-4928-bf81-adcd1981f259"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:40:33.466182 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.466147 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:33.466182 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.466179 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:33.466182 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.466190 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:33.466452 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.466199 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd95eaa-10d3-4928-bf81-adcd1981f259-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:33.466452 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.466210 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:33.466452 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.466218 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:33.466452 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.466227 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd95eaa-10d3-4928-bf81-adcd1981f259-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:33.466452 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.466236 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd95eaa-10d3-4928-bf81-adcd1981f259-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:33.466452 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:33.466245 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6ztx\" (UniqueName: \"kubernetes.io/projected/7bd95eaa-10d3-4928-bf81-adcd1981f259-kube-api-access-v6ztx\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:34.146074 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:34.146037 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" event={"ID":"7bd95eaa-10d3-4928-bf81-adcd1981f259","Type":"ContainerDied","Data":"e926874390dd8327e067caf876ccacb988cf015245c8bbc2b2fe9103c3872f2c"} Nov 02 23:40:34.146074 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:34.146070 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb" Nov 02 23:40:34.146275 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:34.146107 5114 scope.go:122] "RemoveContainer" containerID="85aee179346b724fab77c7db797099ff9ea2790a132a7ac53bee3cd267ec5200" Nov 02 23:40:34.359469 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:34.359433 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb"] Nov 02 23:40:34.364095 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:34.364053 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-85hqb"] Nov 02 23:40:34.793667 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:34.793632 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd95eaa-10d3-4928-bf81-adcd1981f259" path="/var/lib/kubelet/pods/7bd95eaa-10d3-4928-bf81-adcd1981f259/volumes" Nov 02 23:40:37.587155 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:40:37.587114 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:40:48.141650 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.141594 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f"] Nov 02 23:40:48.142142 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.142092 5114 cpu_manager.go:414] "containerMap: removing container" podUID="7bd95eaa-10d3-4928-bf81-adcd1981f259" containerName="istio-proxy" Nov 02 23:40:48.142142 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.142109 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd95eaa-10d3-4928-bf81-adcd1981f259" containerName="istio-proxy" Nov 02 23:40:48.142142 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.142120 5114 cpu_manager.go:414] "containerMap: removing container" podUID="e7d12b96-e660-4927-9424-b5ce36b78ac5" containerName="revoke-keys" Nov 02 23:40:48.142142 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.142125 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7d12b96-e660-4927-9424-b5ce36b78ac5" containerName="revoke-keys" Nov 02 23:40:48.142142 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.142138 5114 cpu_manager.go:414] "containerMap: removing container" podUID="8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5" containerName="maas-api" Nov 02 23:40:48.142142 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.142143 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5" containerName="maas-api" Nov 02 23:40:48.142334 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.142241 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd95eaa-10d3-4928-bf81-adcd1981f259" containerName="istio-proxy" Nov 02 23:40:48.142334 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.142249 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7d12b96-e660-4927-9424-b5ce36b78ac5" containerName="revoke-keys" Nov 02 23:40:48.142334 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.142256 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="8892cb6b-3a9d-4b85-9d48-d4a7f89b78a5" containerName="maas-api" Nov 02 23:40:48.146554 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.146534 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:40:48.242564 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.242508 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4l9b\" (UniqueName: \"kubernetes.io/projected/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-kube-api-access-w4l9b\") pod \"auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f\" (UID: \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:40:48.242767 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.242663 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-tls-certs\") pod \"auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f\" (UID: \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:40:48.242767 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.242736 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-model-cache\") pod \"auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f\" (UID: \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:40:48.242854 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.242812 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-tmp-dir\") pod \"auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f\" (UID: \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:40:48.242854 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.242847 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-home\") pod \"auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f\" (UID: \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:40:48.242940 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.242870 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-dshm\") pod \"auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f\" (UID: \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:40:48.250677 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.250645 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-b8d3cb\"/\"openshift-service-ca.crt\"" Nov 02 23:40:48.251101 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.251079 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-b8d3cb\"/\"kube-root-ca.crt\"" Nov 02 23:40:48.251289 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.251271 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-b8d3cb\"/\"default-dockercfg-mhnqz\"" Nov 02 23:40:48.251356 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.251337 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-b8d3cb\"/\"auth-test-model-b8d3cb-kserve-self-signed-certs\"" Nov 02 23:40:48.253438 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.253415 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f"] Nov 02 23:40:48.344197 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.344157 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4l9b\" (UniqueName: \"kubernetes.io/projected/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-kube-api-access-w4l9b\") pod \"auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f\" (UID: \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:40:48.344432 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.344262 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-tls-certs\") pod \"auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f\" (UID: \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:40:48.344432 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.344346 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-model-cache\") pod \"auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f\" (UID: \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:40:48.344432 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.344394 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-tmp-dir\") pod \"auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f\" (UID: \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:40:48.344432 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.344424 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-home\") pod \"auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f\" (UID: \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:40:48.344729 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.344453 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-dshm\") pod \"auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f\" (UID: \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:40:48.344915 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.344889 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-model-cache\") pod \"auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f\" (UID: \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:40:48.345005 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.344896 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-tmp-dir\") pod \"auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f\" (UID: \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:40:48.345005 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.344954 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-home\") pod \"auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f\" (UID: \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:40:48.347146 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.347114 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-dshm\") pod \"auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f\" (UID: \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:40:48.347654 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.347633 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-tls-certs\") pod \"auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f\" (UID: \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:40:48.405396 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.405315 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4l9b\" (UniqueName: \"kubernetes.io/projected/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-kube-api-access-w4l9b\") pod \"auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f\" (UID: \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:40:48.460176 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.460121 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:40:48.877249 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:48.877212 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f"] Nov 02 23:40:48.881684 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:40:48.881653 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc922368c_26c9_4ed6_9f52_3b7df5f8ce73.slice/crio-c309d29281dfe9e027452205f8ff45899514b883f90729326fda1e0e863740ec WatchSource:0}: Error finding container c309d29281dfe9e027452205f8ff45899514b883f90729326fda1e0e863740ec: Status 404 returned error can't find the container with id c309d29281dfe9e027452205f8ff45899514b883f90729326fda1e0e863740ec Nov 02 23:40:49.232080 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:49.231980 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" event={"ID":"c922368c-26c9-4ed6-9f52-3b7df5f8ce73","Type":"ContainerStarted","Data":"4c83dc4474d2d38bc93f4507ca39dc64311d31a281be3eeb7b311f1c0c6c36dd"} Nov 02 23:40:49.232080 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:49.232031 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" event={"ID":"c922368c-26c9-4ed6-9f52-3b7df5f8ce73","Type":"ContainerStarted","Data":"c309d29281dfe9e027452205f8ff45899514b883f90729326fda1e0e863740ec"} Nov 02 23:40:49.350208 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:49.350146 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" podStartSLOduration=1.350130604 podStartE2EDuration="1.350130604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:40:49.347744774 +0000 UTC m=+3124.700778152" watchObservedRunningTime="2026-11-02 23:40:49.350130604 +0000 UTC m=+3124.703163980" Nov 02 23:40:49.461307 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:49.461268 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:40:58.234660 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.234624 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p"] Nov 02 23:40:58.235180 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.234900 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" podUID="1f195682-c9a3-4e14-994c-5f795c95f41a" containerName="discovery" containerID="cri-o://e9b78df2860d36241278dcce6eb2902385145581eefa8f58b3814d77a2569a47" gracePeriod=30 Nov 02 23:40:58.321634 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.321589 5114 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-cbx4p container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:8080/ready\": dial tcp 10.134.0.93:8080: connect: connection refused" start-of-body= Nov 02 23:40:58.321858 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.321653 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" podUID="1f195682-c9a3-4e14-994c-5f795c95f41a" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.93:8080/ready\": dial tcp 10.134.0.93:8080: connect: connection refused" Nov 02 23:40:58.461352 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.461305 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:40:58.474191 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.474152 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:40:58.510466 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.510440 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:40:58.563597 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.563558 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-csr-dns-cert\") pod \"1f195682-c9a3-4e14-994c-5f795c95f41a\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " Nov 02 23:40:58.563597 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.563600 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f195682-c9a3-4e14-994c-5f795c95f41a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f195682-c9a3-4e14-994c-5f795c95f41a-local-certs\") pod \"1f195682-c9a3-4e14-994c-5f795c95f41a\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " Nov 02 23:40:58.563872 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.563623 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f195682-c9a3-4e14-994c-5f795c95f41a-kube-api-access-n2vnp\" (UniqueName: \"kubernetes.io/projected/1f195682-c9a3-4e14-994c-5f795c95f41a-kube-api-access-n2vnp\") pod \"1f195682-c9a3-4e14-994c-5f795c95f41a\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " Nov 02 23:40:58.563872 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.563652 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f195682-c9a3-4e14-994c-5f795c95f41a-cacerts\" (UniqueName: \"kubernetes.io/secret/1f195682-c9a3-4e14-994c-5f795c95f41a-cacerts\") pod \"1f195682-c9a3-4e14-994c-5f795c95f41a\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " Nov 02 23:40:58.563872 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.563752 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-csr-ca-configmap\") pod \"1f195682-c9a3-4e14-994c-5f795c95f41a\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " Nov 02 23:40:58.564037 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.563872 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-token\" (UniqueName: \"kubernetes.io/projected/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-token\") pod \"1f195682-c9a3-4e14-994c-5f795c95f41a\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " Nov 02 23:40:58.564037 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.563929 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-kubeconfig\") pod \"1f195682-c9a3-4e14-994c-5f795c95f41a\" (UID: \"1f195682-c9a3-4e14-994c-5f795c95f41a\") " Nov 02 23:40:58.565042 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.564634 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-csr-ca-configmap" pod "1f195682-c9a3-4e14-994c-5f795c95f41a" (UID: "1f195682-c9a3-4e14-994c-5f795c95f41a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:40:58.566446 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.566416 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f195682-c9a3-4e14-994c-5f795c95f41a-cacerts" pod "1f195682-c9a3-4e14-994c-5f795c95f41a" (UID: "1f195682-c9a3-4e14-994c-5f795c95f41a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:40:58.566446 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.566437 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-kubeconfig" pod "1f195682-c9a3-4e14-994c-5f795c95f41a" (UID: "1f195682-c9a3-4e14-994c-5f795c95f41a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:40:58.566721 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.566700 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f195682-c9a3-4e14-994c-5f795c95f41a-local-certs" pod "1f195682-c9a3-4e14-994c-5f795c95f41a" (UID: "1f195682-c9a3-4e14-994c-5f795c95f41a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:40:58.566792 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.566720 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f195682-c9a3-4e14-994c-5f795c95f41a-kube-api-access-n2vnp" pod "1f195682-c9a3-4e14-994c-5f795c95f41a" (UID: "1f195682-c9a3-4e14-994c-5f795c95f41a"). InnerVolumeSpecName "kube-api-access-n2vnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:40:58.566792 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.566783 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-csr-dns-cert" pod "1f195682-c9a3-4e14-994c-5f795c95f41a" (UID: "1f195682-c9a3-4e14-994c-5f795c95f41a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:40:58.566922 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.566905 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-token" pod "1f195682-c9a3-4e14-994c-5f795c95f41a" (UID: "1f195682-c9a3-4e14-994c-5f795c95f41a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:40:58.665502 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.665456 5114 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-kubeconfig\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:58.665694 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.665521 5114 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-csr-dns-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:58.665694 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.665533 5114 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f195682-c9a3-4e14-994c-5f795c95f41a-local-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:58.665694 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.665542 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2vnp\" (UniqueName: \"kubernetes.io/projected/1f195682-c9a3-4e14-994c-5f795c95f41a-kube-api-access-n2vnp\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:58.665694 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.665552 5114 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1f195682-c9a3-4e14-994c-5f795c95f41a-cacerts\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:58.665694 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.665563 5114 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-csr-ca-configmap\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:58.665694 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:58.665571 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f195682-c9a3-4e14-994c-5f795c95f41a-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:59.292352 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:59.292317 5114 generic.go:356] "Generic (PLEG): container finished" podID="1f195682-c9a3-4e14-994c-5f795c95f41a" containerID="e9b78df2860d36241278dcce6eb2902385145581eefa8f58b3814d77a2569a47" exitCode=0 Nov 02 23:40:59.292837 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:59.292393 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" Nov 02 23:40:59.292837 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:59.292399 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" event={"ID":"1f195682-c9a3-4e14-994c-5f795c95f41a","Type":"ContainerDied","Data":"e9b78df2860d36241278dcce6eb2902385145581eefa8f58b3814d77a2569a47"} Nov 02 23:40:59.292837 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:59.292460 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p" event={"ID":"1f195682-c9a3-4e14-994c-5f795c95f41a","Type":"ContainerDied","Data":"a53bfa778de53b1787d4dc260e8be7f7082e69b354d107f9d331ffc7c8e16805"} Nov 02 23:40:59.292837 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:59.292504 5114 scope.go:122] "RemoveContainer" containerID="e9b78df2860d36241278dcce6eb2902385145581eefa8f58b3814d77a2569a47" Nov 02 23:40:59.303337 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:59.303289 5114 scope.go:122] "RemoveContainer" containerID="e9b78df2860d36241278dcce6eb2902385145581eefa8f58b3814d77a2569a47" Nov 02 23:40:59.303879 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:40:59.303851 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9b78df2860d36241278dcce6eb2902385145581eefa8f58b3814d77a2569a47\": container with ID starting with e9b78df2860d36241278dcce6eb2902385145581eefa8f58b3814d77a2569a47 not found: ID does not exist" containerID="e9b78df2860d36241278dcce6eb2902385145581eefa8f58b3814d77a2569a47" Nov 02 23:40:59.303999 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:59.303899 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9b78df2860d36241278dcce6eb2902385145581eefa8f58b3814d77a2569a47"} err="failed to get container status \"e9b78df2860d36241278dcce6eb2902385145581eefa8f58b3814d77a2569a47\": rpc error: code = NotFound desc = could not find container \"e9b78df2860d36241278dcce6eb2902385145581eefa8f58b3814d77a2569a47\": container with ID starting with e9b78df2860d36241278dcce6eb2902385145581eefa8f58b3814d77a2569a47 not found: ID does not exist" Nov 02 23:40:59.306236 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:59.306212 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:40:59.506759 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:59.506715 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p"] Nov 02 23:40:59.510699 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:40:59.510663 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cbx4p"] Nov 02 23:41:00.794398 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:00.794363 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f195682-c9a3-4e14-994c-5f795c95f41a" path="/var/lib/kubelet/pods/1f195682-c9a3-4e14-994c-5f795c95f41a/volumes" Nov 02 23:41:04.995288 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:04.995248 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c516707-ds7zn"] Nov 02 23:41:04.995870 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:04.995842 5114 cpu_manager.go:414] "containerMap: removing container" podUID="1f195682-c9a3-4e14-994c-5f795c95f41a" containerName="discovery" Nov 02 23:41:04.995870 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:04.995866 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f195682-c9a3-4e14-994c-5f795c95f41a" containerName="discovery" Nov 02 23:41:04.995975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:04.995934 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f195682-c9a3-4e14-994c-5f795c95f41a" containerName="discovery" Nov 02 23:41:05.000219 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:05.000196 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c516707-ds7zn" Nov 02 23:41:05.052415 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:05.052382 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:41:05.106041 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:05.105999 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c516707-ds7zn"] Nov 02 23:41:05.106211 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:05.106155 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:41:05.137784 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:05.137749 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qww86\" (UniqueName: \"kubernetes.io/projected/02495134-1dc0-44f6-a856-5c8f8d2ec72c-kube-api-access-qww86\") pod \"maas-api-revoke-sub-6c516707-ds7zn\" (UID: \"02495134-1dc0-44f6-a856-5c8f8d2ec72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c516707-ds7zn" Nov 02 23:41:05.137784 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:05.137792 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02495134-1dc0-44f6-a856-5c8f8d2ec72c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c516707-ds7zn\" (UID: \"02495134-1dc0-44f6-a856-5c8f8d2ec72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c516707-ds7zn" Nov 02 23:41:05.138035 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:05.137939 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02495134-1dc0-44f6-a856-5c8f8d2ec72c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c516707-ds7zn\" (UID: \"02495134-1dc0-44f6-a856-5c8f8d2ec72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c516707-ds7zn" Nov 02 23:41:05.238736 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:05.238692 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qww86\" (UniqueName: \"kubernetes.io/projected/02495134-1dc0-44f6-a856-5c8f8d2ec72c-kube-api-access-qww86\") pod \"maas-api-revoke-sub-6c516707-ds7zn\" (UID: \"02495134-1dc0-44f6-a856-5c8f8d2ec72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c516707-ds7zn" Nov 02 23:41:05.238736 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:05.238740 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02495134-1dc0-44f6-a856-5c8f8d2ec72c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c516707-ds7zn\" (UID: \"02495134-1dc0-44f6-a856-5c8f8d2ec72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c516707-ds7zn" Nov 02 23:41:05.238968 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:05.238800 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02495134-1dc0-44f6-a856-5c8f8d2ec72c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c516707-ds7zn\" (UID: \"02495134-1dc0-44f6-a856-5c8f8d2ec72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c516707-ds7zn" Nov 02 23:41:05.239632 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:05.239603 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02495134-1dc0-44f6-a856-5c8f8d2ec72c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c516707-ds7zn\" (UID: \"02495134-1dc0-44f6-a856-5c8f8d2ec72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c516707-ds7zn" Nov 02 23:41:05.241399 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:05.241370 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02495134-1dc0-44f6-a856-5c8f8d2ec72c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c516707-ds7zn\" (UID: \"02495134-1dc0-44f6-a856-5c8f8d2ec72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c516707-ds7zn" Nov 02 23:41:05.300101 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:05.300061 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qww86\" (UniqueName: \"kubernetes.io/projected/02495134-1dc0-44f6-a856-5c8f8d2ec72c-kube-api-access-qww86\") pod \"maas-api-revoke-sub-6c516707-ds7zn\" (UID: \"02495134-1dc0-44f6-a856-5c8f8d2ec72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c516707-ds7zn" Nov 02 23:41:05.314846 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:05.314816 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c516707-ds7zn" Nov 02 23:41:05.690841 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:05.690812 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c516707-ds7zn"] Nov 02 23:41:05.693554 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:41:05.693522 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02495134_1dc0_44f6_a856_5c8f8d2ec72c.slice/crio-ac88e006b4f392505a0bf652a4859a48f4858724b92d8666bf760dc1a623900e WatchSource:0}: Error finding container ac88e006b4f392505a0bf652a4859a48f4858724b92d8666bf760dc1a623900e: Status 404 returned error can't find the container with id ac88e006b4f392505a0bf652a4859a48f4858724b92d8666bf760dc1a623900e Nov 02 23:41:06.334449 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:06.334405 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c516707-ds7zn" event={"ID":"02495134-1dc0-44f6-a856-5c8f8d2ec72c","Type":"ContainerStarted","Data":"2cec59850e5a59806f801c9ba85162e76c794cab7a2c913048163923cfa6d454"} Nov 02 23:41:06.334942 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:06.334469 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c516707-ds7zn" event={"ID":"02495134-1dc0-44f6-a856-5c8f8d2ec72c","Type":"ContainerStarted","Data":"ac88e006b4f392505a0bf652a4859a48f4858724b92d8666bf760dc1a623900e"} Nov 02 23:41:06.450424 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:06.450371 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c516707-ds7zn" podStartSLOduration=2.450353938 podStartE2EDuration="2.450353938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:41:06.449063693 +0000 UTC m=+3141.802097070" watchObservedRunningTime="2026-11-02 23:41:06.450353938 +0000 UTC m=+3141.803387315" Nov 02 23:41:07.341249 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:07.341211 5114 generic.go:356] "Generic (PLEG): container finished" podID="02495134-1dc0-44f6-a856-5c8f8d2ec72c" containerID="2cec59850e5a59806f801c9ba85162e76c794cab7a2c913048163923cfa6d454" exitCode=0 Nov 02 23:41:07.341725 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:07.341299 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c516707-ds7zn" event={"ID":"02495134-1dc0-44f6-a856-5c8f8d2ec72c","Type":"ContainerDied","Data":"2cec59850e5a59806f801c9ba85162e76c794cab7a2c913048163923cfa6d454"} Nov 02 23:41:07.586045 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:41:07.586005 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:41:08.499534 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:08.499505 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c516707-ds7zn" Nov 02 23:41:08.577882 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:08.577845 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02495134-1dc0-44f6-a856-5c8f8d2ec72c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02495134-1dc0-44f6-a856-5c8f8d2ec72c-maas-api-openshift-ca\") pod \"02495134-1dc0-44f6-a856-5c8f8d2ec72c\" (UID: \"02495134-1dc0-44f6-a856-5c8f8d2ec72c\") " Nov 02 23:41:08.578088 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:08.577936 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02495134-1dc0-44f6-a856-5c8f8d2ec72c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02495134-1dc0-44f6-a856-5c8f8d2ec72c-maas-api-xks-ca\") pod \"02495134-1dc0-44f6-a856-5c8f8d2ec72c\" (UID: \"02495134-1dc0-44f6-a856-5c8f8d2ec72c\") " Nov 02 23:41:08.578088 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:08.578003 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02495134-1dc0-44f6-a856-5c8f8d2ec72c-kube-api-access-qww86\" (UniqueName: \"kubernetes.io/projected/02495134-1dc0-44f6-a856-5c8f8d2ec72c-kube-api-access-qww86\") pod \"02495134-1dc0-44f6-a856-5c8f8d2ec72c\" (UID: \"02495134-1dc0-44f6-a856-5c8f8d2ec72c\") " Nov 02 23:41:08.578294 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:08.578266 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02495134-1dc0-44f6-a856-5c8f8d2ec72c-maas-api-openshift-ca" pod "02495134-1dc0-44f6-a856-5c8f8d2ec72c" (UID: "02495134-1dc0-44f6-a856-5c8f8d2ec72c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:41:08.578442 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:08.578413 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02495134-1dc0-44f6-a856-5c8f8d2ec72c-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:08.580523 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:08.580466 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02495134-1dc0-44f6-a856-5c8f8d2ec72c-kube-api-access-qww86" pod "02495134-1dc0-44f6-a856-5c8f8d2ec72c" (UID: "02495134-1dc0-44f6-a856-5c8f8d2ec72c"). InnerVolumeSpecName "kube-api-access-qww86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:41:08.580646 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:08.580562 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02495134-1dc0-44f6-a856-5c8f8d2ec72c-maas-api-xks-ca" pod "02495134-1dc0-44f6-a856-5c8f8d2ec72c" (UID: "02495134-1dc0-44f6-a856-5c8f8d2ec72c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:41:08.679426 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:08.679329 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02495134-1dc0-44f6-a856-5c8f8d2ec72c-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:08.679426 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:08.679367 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qww86\" (UniqueName: \"kubernetes.io/projected/02495134-1dc0-44f6-a856-5c8f8d2ec72c-kube-api-access-qww86\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:09.353308 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:09.353274 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c516707-ds7zn" Nov 02 23:41:09.353537 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:09.353308 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c516707-ds7zn" event={"ID":"02495134-1dc0-44f6-a856-5c8f8d2ec72c","Type":"ContainerDied","Data":"ac88e006b4f392505a0bf652a4859a48f4858724b92d8666bf760dc1a623900e"} Nov 02 23:41:09.353537 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:09.353361 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac88e006b4f392505a0bf652a4859a48f4858724b92d8666bf760dc1a623900e" Nov 02 23:41:14.346993 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.346955 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5"] Nov 02 23:41:14.347616 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.347594 5114 cpu_manager.go:414] "containerMap: removing container" podUID="02495134-1dc0-44f6-a856-5c8f8d2ec72c" containerName="revoke-keys" Nov 02 23:41:14.347689 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.347620 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02495134-1dc0-44f6-a856-5c8f8d2ec72c" containerName="revoke-keys" Nov 02 23:41:14.347776 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.347764 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="02495134-1dc0-44f6-a856-5c8f8d2ec72c" containerName="revoke-keys" Nov 02 23:41:14.351319 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.351300 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.404193 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.403523 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-875049-openshift-default-dockercfg-hprqp\"" Nov 02 23:41:14.446164 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.446127 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-envoy\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.446346 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.446191 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fkrf\" (UniqueName: \"kubernetes.io/projected/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-kube-api-access-7fkrf\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.446346 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.446248 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-podinfo\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.446346 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.446289 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istiod-ca-cert\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.446509 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.446439 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-data\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.446557 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.446535 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-workload-certs\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.446639 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.446617 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-credential-socket\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.446787 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.446678 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-workload-socket\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.446787 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.446712 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-token\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.457955 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.457923 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5"] Nov 02 23:41:14.547956 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.547912 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-data\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.548161 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.547962 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-workload-certs\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.548161 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.548025 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-credential-socket\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.548161 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.548077 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-workload-socket\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.548161 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.548103 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-token\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.548391 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.548179 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-envoy\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.548391 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.548217 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fkrf\" (UniqueName: \"kubernetes.io/projected/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-kube-api-access-7fkrf\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.548391 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.548234 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-podinfo\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.548391 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.548255 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istiod-ca-cert\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.548622 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.548402 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-data\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.548622 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.548465 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-workload-certs\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.548622 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.548519 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-credential-socket\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.549114 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.548791 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-workload-socket\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.549114 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.549035 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istiod-ca-cert\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.551737 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.551711 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-envoy\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.552055 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.552031 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-podinfo\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.609537 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.609447 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-token\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.609764 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.609739 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fkrf\" (UniqueName: \"kubernetes.io/projected/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-kube-api-access-7fkrf\") pod \"e2e-autores-875049-openshift-default-574965c6f-jzzx5\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:14.665893 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:14.665862 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:15.105328 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:15.105078 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5"] Nov 02 23:41:15.105761 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:41:15.105724 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa21ce01_b481_4cd5_bbba_8e00e9f9769d.slice/crio-25efc2fdcb753875f4891b29aaf0abb1bec7ca0555cf56bde6cfacbd9c72fd0b WatchSource:0}: Error finding container 25efc2fdcb753875f4891b29aaf0abb1bec7ca0555cf56bde6cfacbd9c72fd0b: Status 404 returned error can't find the container with id 25efc2fdcb753875f4891b29aaf0abb1bec7ca0555cf56bde6cfacbd9c72fd0b Nov 02 23:41:15.110353 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:15.110315 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:41:15.110522 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:15.110451 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:41:15.110853 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:15.110819 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:41:15.391873 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:15.391778 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" event={"ID":"fa21ce01-b481-4cd5-bbba-8e00e9f9769d","Type":"ContainerStarted","Data":"ddf7ba4f39f9b0ce871e7dc4047d4bcbaed49ae4eb612205e29296a9ce5deae0"} Nov 02 23:41:15.391873 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:15.391843 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" event={"ID":"fa21ce01-b481-4cd5-bbba-8e00e9f9769d","Type":"ContainerStarted","Data":"25efc2fdcb753875f4891b29aaf0abb1bec7ca0555cf56bde6cfacbd9c72fd0b"} Nov 02 23:41:15.515071 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:15.515012 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" podStartSLOduration=1.51499623 podStartE2EDuration="1.51499623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:41:15.511172103 +0000 UTC m=+3150.864205479" watchObservedRunningTime="2026-11-02 23:41:15.51499623 +0000 UTC m=+3150.868029604" Nov 02 23:41:15.666871 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:15.666780 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:16.672698 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:16.672666 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:17.403264 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:17.403229 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:17.404249 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:17.404227 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:41:22.144829 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:22.144793 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k"] Nov 02 23:41:22.148741 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:22.148708 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k" Nov 02 23:41:22.204591 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:22.204557 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k"] Nov 02 23:41:22.238193 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:22.238147 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw4wj\" (UniqueName: \"kubernetes.io/projected/fcdda5bf-a52f-427a-b68e-9aa605f6682a-kube-api-access-pw4wj\") pod \"payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k\" (UID: \"fcdda5bf-a52f-427a-b68e-9aa605f6682a\") " pod="openshift-ingress/payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k" Nov 02 23:41:22.238193 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:22.238192 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcdda5bf-a52f-427a-b68e-9aa605f6682a-plugins-config-volume\") pod \"payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k\" (UID: \"fcdda5bf-a52f-427a-b68e-9aa605f6682a\") " pod="openshift-ingress/payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k" Nov 02 23:41:22.251098 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:22.251062 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-875049\"" Nov 02 23:41:22.251098 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:22.251100 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-875049-dockercfg-k646k\"" Nov 02 23:41:22.339311 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:22.339278 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw4wj\" (UniqueName: \"kubernetes.io/projected/fcdda5bf-a52f-427a-b68e-9aa605f6682a-kube-api-access-pw4wj\") pod \"payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k\" (UID: \"fcdda5bf-a52f-427a-b68e-9aa605f6682a\") " pod="openshift-ingress/payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k" Nov 02 23:41:22.339311 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:22.339314 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcdda5bf-a52f-427a-b68e-9aa605f6682a-plugins-config-volume\") pod \"payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k\" (UID: \"fcdda5bf-a52f-427a-b68e-9aa605f6682a\") " pod="openshift-ingress/payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k" Nov 02 23:41:22.340086 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:22.340067 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcdda5bf-a52f-427a-b68e-9aa605f6682a-plugins-config-volume\") pod \"payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k\" (UID: \"fcdda5bf-a52f-427a-b68e-9aa605f6682a\") " pod="openshift-ingress/payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k" Nov 02 23:41:22.400619 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:22.400543 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw4wj\" (UniqueName: \"kubernetes.io/projected/fcdda5bf-a52f-427a-b68e-9aa605f6682a-kube-api-access-pw4wj\") pod \"payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k\" (UID: \"fcdda5bf-a52f-427a-b68e-9aa605f6682a\") " pod="openshift-ingress/payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k" Nov 02 23:41:22.465233 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:22.465190 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k" Nov 02 23:41:22.844746 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:22.844713 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k"] Nov 02 23:41:22.846873 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:41:22.846841 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcdda5bf_a52f_427a_b68e_9aa605f6682a.slice/crio-79355e6b02008e5571fa1cb0390f3b3db8d75c77a2c561a465d48924bad1367f WatchSource:0}: Error finding container 79355e6b02008e5571fa1cb0390f3b3db8d75c77a2c561a465d48924bad1367f: Status 404 returned error can't find the container with id 79355e6b02008e5571fa1cb0390f3b3db8d75c77a2c561a465d48924bad1367f Nov 02 23:41:22.848911 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:22.848886 5114 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:41:23.437287 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:23.437252 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k" event={"ID":"fcdda5bf-a52f-427a-b68e-9aa605f6682a","Type":"ContainerStarted","Data":"79355e6b02008e5571fa1cb0390f3b3db8d75c77a2c561a465d48924bad1367f"} Nov 02 23:41:26.238993 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:26.238957 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm"] Nov 02 23:41:26.239462 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:26.239184 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" podUID="d05d07a8-a317-42d7-b48c-7be5d5fed71a" containerName="istio-proxy" containerID="cri-o://dbf7f411da1fdb0a2cc651c98cdf266e1c274e2b70d6fb156ec9f336a551d90b" gracePeriod=30 Nov 02 23:41:26.456514 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:26.456456 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k" event={"ID":"fcdda5bf-a52f-427a-b68e-9aa605f6682a","Type":"ContainerStarted","Data":"150cc2f3be906200737cb4e0440f0a079ab6f5362023975e88e27b2ab80dc0c3"} Nov 02 23:41:26.456706 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:26.456547 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k" Nov 02 23:41:26.572987 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:26.572914 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k" podStartSLOduration=1.869786183 podStartE2EDuration="4.572898914s" totalImagesPullingTime="2.703112731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:41:26.570673627 +0000 UTC m=+3161.923707088" watchObservedRunningTime="2026-11-02 23:41:26.572898914 +0000 UTC m=+3161.925932346" Nov 02 23:41:31.486265 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.486229 5114 generic.go:356] "Generic (PLEG): container finished" podID="d05d07a8-a317-42d7-b48c-7be5d5fed71a" containerID="dbf7f411da1fdb0a2cc651c98cdf266e1c274e2b70d6fb156ec9f336a551d90b" exitCode=0 Nov 02 23:41:31.486668 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.486300 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" event={"ID":"d05d07a8-a317-42d7-b48c-7be5d5fed71a","Type":"ContainerDied","Data":"dbf7f411da1fdb0a2cc651c98cdf266e1c274e2b70d6fb156ec9f336a551d90b"} Nov 02 23:41:31.486668 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.486380 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" event={"ID":"d05d07a8-a317-42d7-b48c-7be5d5fed71a","Type":"ContainerDied","Data":"d448b60da2a0e1cff16e3a2ae9fab8a81c26d8a9dcbd3b2a477f3997274f50c1"} Nov 02 23:41:31.486668 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.486407 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d448b60da2a0e1cff16e3a2ae9fab8a81c26d8a9dcbd3b2a477f3997274f50c1" Nov 02 23:41:31.503413 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.503382 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:41:31.545104 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.545064 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istiod-ca-cert\") pod \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " Nov 02 23:41:31.545318 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.545171 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-workload-certs\") pod \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " Nov 02 23:41:31.545318 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.545203 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-data\") pod \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " Nov 02 23:41:31.545318 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.545244 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d05d07a8-a317-42d7-b48c-7be5d5fed71a-kube-api-access-jrl5h\" (UniqueName: \"kubernetes.io/projected/d05d07a8-a317-42d7-b48c-7be5d5fed71a-kube-api-access-jrl5h\") pod \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " Nov 02 23:41:31.545318 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.545299 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-credential-socket\") pod \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " Nov 02 23:41:31.545563 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.545405 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-envoy\") pod \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " Nov 02 23:41:31.545563 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.545535 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-token\" (UniqueName: \"kubernetes.io/projected/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-token\") pod \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " Nov 02 23:41:31.545955 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.545547 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-workload-certs" pod "d05d07a8-a317-42d7-b48c-7be5d5fed71a" (UID: "d05d07a8-a317-42d7-b48c-7be5d5fed71a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:41:31.545955 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.545563 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-data" pod "d05d07a8-a317-42d7-b48c-7be5d5fed71a" (UID: "d05d07a8-a317-42d7-b48c-7be5d5fed71a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:41:31.545955 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.545569 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-workload-socket\") pod \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " Nov 02 23:41:31.545955 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.545711 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-podinfo\") pod \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\" (UID: \"d05d07a8-a317-42d7-b48c-7be5d5fed71a\") " Nov 02 23:41:31.545955 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.545583 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istiod-ca-cert" pod "d05d07a8-a317-42d7-b48c-7be5d5fed71a" (UID: "d05d07a8-a317-42d7-b48c-7be5d5fed71a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:41:31.545955 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.545698 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-credential-socket" pod "d05d07a8-a317-42d7-b48c-7be5d5fed71a" (UID: "d05d07a8-a317-42d7-b48c-7be5d5fed71a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:41:31.545955 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.545770 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-workload-socket" pod "d05d07a8-a317-42d7-b48c-7be5d5fed71a" (UID: "d05d07a8-a317-42d7-b48c-7be5d5fed71a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:41:31.546530 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.546160 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:31.546530 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.546178 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:31.546530 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.546187 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:31.546530 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.546197 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:31.546530 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.546205 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:31.548122 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.548092 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-token" pod "d05d07a8-a317-42d7-b48c-7be5d5fed71a" (UID: "d05d07a8-a317-42d7-b48c-7be5d5fed71a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:41:31.548266 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.548233 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-envoy" pod "d05d07a8-a317-42d7-b48c-7be5d5fed71a" (UID: "d05d07a8-a317-42d7-b48c-7be5d5fed71a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:41:31.548556 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.548529 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-podinfo" pod "d05d07a8-a317-42d7-b48c-7be5d5fed71a" (UID: "d05d07a8-a317-42d7-b48c-7be5d5fed71a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:41:31.548556 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.548537 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05d07a8-a317-42d7-b48c-7be5d5fed71a-kube-api-access-jrl5h" pod "d05d07a8-a317-42d7-b48c-7be5d5fed71a" (UID: "d05d07a8-a317-42d7-b48c-7be5d5fed71a"). InnerVolumeSpecName "kube-api-access-jrl5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:41:31.647069 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.647012 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrl5h\" (UniqueName: \"kubernetes.io/projected/d05d07a8-a317-42d7-b48c-7be5d5fed71a-kube-api-access-jrl5h\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:31.647069 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.647060 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:31.647069 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.647071 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:31.647330 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:31.647093 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05d07a8-a317-42d7-b48c-7be5d5fed71a-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:41:32.491238 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:32.491205 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm" Nov 02 23:41:32.705646 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:32.705606 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm"] Nov 02 23:41:32.755248 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:32.755151 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2d2hm"] Nov 02 23:41:32.793216 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:32.793181 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d05d07a8-a317-42d7-b48c-7be5d5fed71a" path="/var/lib/kubelet/pods/d05d07a8-a317-42d7-b48c-7be5d5fed71a/volumes" Nov 02 23:41:37.466430 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:37.466397 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k" Nov 02 23:41:37.586047 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:41:37.586011 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:41:38.999718 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:38.999678 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst"] Nov 02 23:41:39.000143 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.000072 5114 cpu_manager.go:414] "containerMap: removing container" podUID="d05d07a8-a317-42d7-b48c-7be5d5fed71a" containerName="istio-proxy" Nov 02 23:41:39.000143 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.000083 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d05d07a8-a317-42d7-b48c-7be5d5fed71a" containerName="istio-proxy" Nov 02 23:41:39.000220 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.000199 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="d05d07a8-a317-42d7-b48c-7be5d5fed71a" containerName="istio-proxy" Nov 02 23:41:39.004617 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.004592 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.056686 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.056647 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b42983-openshift-default-dockercfg-xchbk\"" Nov 02 23:41:39.109296 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.109261 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst"] Nov 02 23:41:39.127655 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.127606 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.127876 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.127690 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.127876 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.127791 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.127876 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.127858 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.128019 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.127896 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb85v\" (UniqueName: \"kubernetes.io/projected/fa812057-3300-4df2-acf5-59ac9fd4999e-kube-api-access-vb85v\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.128019 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.127948 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.128019 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.128003 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.128157 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.128041 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa812057-3300-4df2-acf5-59ac9fd4999e-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.128157 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.128075 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.229874 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.229830 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.229874 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.229876 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb85v\" (UniqueName: \"kubernetes.io/projected/fa812057-3300-4df2-acf5-59ac9fd4999e-kube-api-access-vb85v\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.230144 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.229913 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.230144 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.229943 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.230144 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.229962 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa812057-3300-4df2-acf5-59ac9fd4999e-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.230144 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.229985 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.230144 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.230061 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.230144 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.230099 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.230466 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.230195 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.230466 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.230435 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.230671 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.230596 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.230671 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.230636 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.230828 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.230807 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa812057-3300-4df2-acf5-59ac9fd4999e-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.230993 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.230969 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.232502 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.232460 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.232855 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.232835 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.291146 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.291107 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.291290 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.291178 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb85v\" (UniqueName: \"kubernetes.io/projected/fa812057-3300-4df2-acf5-59ac9fd4999e-kube-api-access-vb85v\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-npfst\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.318868 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.318835 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:39.754615 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.754583 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst"] Nov 02 23:41:39.756440 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:41:39.756412 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa812057_3300_4df2_acf5_59ac9fd4999e.slice/crio-22e2683cc8e2889a46e31239dee05cbf238d3b3514cecbae4cc34c84df04bd2e WatchSource:0}: Error finding container 22e2683cc8e2889a46e31239dee05cbf238d3b3514cecbae4cc34c84df04bd2e: Status 404 returned error can't find the container with id 22e2683cc8e2889a46e31239dee05cbf238d3b3514cecbae4cc34c84df04bd2e Nov 02 23:41:39.759095 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.759061 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:41:39.759172 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.759130 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:41:39.759172 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:39.759162 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:41:40.537395 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:40.537360 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" event={"ID":"fa812057-3300-4df2-acf5-59ac9fd4999e","Type":"ContainerStarted","Data":"0465d92a44345b389d3f81bb52d879e6ebc50d222ce1605fd762c2f63c95fe3b"} Nov 02 23:41:40.539571 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:40.537414 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" event={"ID":"fa812057-3300-4df2-acf5-59ac9fd4999e","Type":"ContainerStarted","Data":"22e2683cc8e2889a46e31239dee05cbf238d3b3514cecbae4cc34c84df04bd2e"} Nov 02 23:41:40.721325 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:40.721248 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" podStartSLOduration=2.721226535 podStartE2EDuration="2.721226535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:41:40.706054083 +0000 UTC m=+3176.059087459" watchObservedRunningTime="2026-11-02 23:41:40.721226535 +0000 UTC m=+3176.074259911" Nov 02 23:41:41.326509 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:41.325881 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:41.330672 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:41.330638 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-npfst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 23:41:41.330868 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:41.330694 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" podUID="fa812057-3300-4df2-acf5-59ac9fd4999e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 23:41:42.331438 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:42.331396 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-npfst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 23:41:42.332033 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:42.331465 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" podUID="fa812057-3300-4df2-acf5-59ac9fd4999e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 23:41:43.330023 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:43.329982 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-npfst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 23:41:43.330336 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:43.330303 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" podUID="fa812057-3300-4df2-acf5-59ac9fd4999e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 23:41:44.330854 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:44.330806 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-npfst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 23:41:44.331272 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:44.330898 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" podUID="fa812057-3300-4df2-acf5-59ac9fd4999e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 23:41:45.327508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:45.326943 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-npfst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 23:41:45.327508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:45.327053 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" podUID="fa812057-3300-4df2-acf5-59ac9fd4999e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 23:41:45.592194 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:45.590227 5114 scope.go:122] "RemoveContainer" containerID="dbf7f411da1fdb0a2cc651c98cdf266e1c274e2b70d6fb156ec9f336a551d90b" Nov 02 23:41:46.320296 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:46.320259 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-npfst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 23:41:46.320652 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:46.320329 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" podUID="fa812057-3300-4df2-acf5-59ac9fd4999e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 23:41:47.324637 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:47.324603 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:47.325064 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:47.324877 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:41:47.325701 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:41:47.325683 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:42:07.588756 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:42:07.588716 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:42:37.585264 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:42:37.585225 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:42:46.654218 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:46.654174 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr"] Nov 02 23:42:46.654788 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:46.654514 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" podUID="191ad0c4-d290-448b-99ec-9304cd39778c" containerName="istio-proxy" containerID="cri-o://657ab192bb55bfa4e7adcf59baafa53da6d9e2f18e8f6eb04329682a35d97237" gracePeriod=30 Nov 02 23:42:49.799180 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:49.799138 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:42:49.799706 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:49.799203 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" podUID="191ad0c4-d290-448b-99ec-9304cd39778c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:42:51.928748 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:51.928679 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:42:51.937643 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:51.937609 5114 generic.go:356] "Generic (PLEG): container finished" podID="191ad0c4-d290-448b-99ec-9304cd39778c" containerID="657ab192bb55bfa4e7adcf59baafa53da6d9e2f18e8f6eb04329682a35d97237" exitCode=0 Nov 02 23:42:51.937820 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:51.937665 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" Nov 02 23:42:51.937820 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:51.937675 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" event={"ID":"191ad0c4-d290-448b-99ec-9304cd39778c","Type":"ContainerDied","Data":"657ab192bb55bfa4e7adcf59baafa53da6d9e2f18e8f6eb04329682a35d97237"} Nov 02 23:42:51.937820 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:51.937764 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr" event={"ID":"191ad0c4-d290-448b-99ec-9304cd39778c","Type":"ContainerDied","Data":"b2ac5e1c10e31d3ba3953194e40e3acdc886d4ec0ea06970541959084e84fcb7"} Nov 02 23:42:51.937820 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:51.937788 5114 scope.go:122] "RemoveContainer" containerID="657ab192bb55bfa4e7adcf59baafa53da6d9e2f18e8f6eb04329682a35d97237" Nov 02 23:42:51.948180 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:51.948156 5114 scope.go:122] "RemoveContainer" containerID="657ab192bb55bfa4e7adcf59baafa53da6d9e2f18e8f6eb04329682a35d97237" Nov 02 23:42:51.948576 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:42:51.948555 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"657ab192bb55bfa4e7adcf59baafa53da6d9e2f18e8f6eb04329682a35d97237\": container with ID starting with 657ab192bb55bfa4e7adcf59baafa53da6d9e2f18e8f6eb04329682a35d97237 not found: ID does not exist" containerID="657ab192bb55bfa4e7adcf59baafa53da6d9e2f18e8f6eb04329682a35d97237" Nov 02 23:42:51.948652 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:51.948588 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657ab192bb55bfa4e7adcf59baafa53da6d9e2f18e8f6eb04329682a35d97237"} err="failed to get container status \"657ab192bb55bfa4e7adcf59baafa53da6d9e2f18e8f6eb04329682a35d97237\": rpc error: code = NotFound desc = could not find container \"657ab192bb55bfa4e7adcf59baafa53da6d9e2f18e8f6eb04329682a35d97237\": container with ID starting with 657ab192bb55bfa4e7adcf59baafa53da6d9e2f18e8f6eb04329682a35d97237 not found: ID does not exist" Nov 02 23:42:52.086246 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.086207 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/191ad0c4-d290-448b-99ec-9304cd39778c-kube-api-access-t86tn\" (UniqueName: \"kubernetes.io/projected/191ad0c4-d290-448b-99ec-9304cd39778c-kube-api-access-t86tn\") pod \"191ad0c4-d290-448b-99ec-9304cd39778c\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " Nov 02 23:42:52.086454 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.086257 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-workload-certs\") pod \"191ad0c4-d290-448b-99ec-9304cd39778c\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " Nov 02 23:42:52.086454 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.086292 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/191ad0c4-d290-448b-99ec-9304cd39778c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191ad0c4-d290-448b-99ec-9304cd39778c-istiod-ca-cert\") pod \"191ad0c4-d290-448b-99ec-9304cd39778c\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " Nov 02 23:42:52.086454 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.086324 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-workload-socket\") pod \"191ad0c4-d290-448b-99ec-9304cd39778c\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " Nov 02 23:42:52.086454 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.086367 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-credential-socket\") pod \"191ad0c4-d290-448b-99ec-9304cd39778c\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " Nov 02 23:42:52.086454 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.086427 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-istio-envoy\") pod \"191ad0c4-d290-448b-99ec-9304cd39778c\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " Nov 02 23:42:52.086812 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.086467 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/191ad0c4-d290-448b-99ec-9304cd39778c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191ad0c4-d290-448b-99ec-9304cd39778c-istio-podinfo\") pod \"191ad0c4-d290-448b-99ec-9304cd39778c\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " Nov 02 23:42:52.086812 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.086603 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/191ad0c4-d290-448b-99ec-9304cd39778c-istio-token\" (UniqueName: \"kubernetes.io/projected/191ad0c4-d290-448b-99ec-9304cd39778c-istio-token\") pod \"191ad0c4-d290-448b-99ec-9304cd39778c\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " Nov 02 23:42:52.086812 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.086586 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-workload-certs" pod "191ad0c4-d290-448b-99ec-9304cd39778c" (UID: "191ad0c4-d290-448b-99ec-9304cd39778c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:42:52.086812 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.086692 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-istio-data\") pod \"191ad0c4-d290-448b-99ec-9304cd39778c\" (UID: \"191ad0c4-d290-448b-99ec-9304cd39778c\") " Nov 02 23:42:52.087035 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.086821 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-credential-socket" pod "191ad0c4-d290-448b-99ec-9304cd39778c" (UID: "191ad0c4-d290-448b-99ec-9304cd39778c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:42:52.087097 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.087056 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-istio-data" pod "191ad0c4-d290-448b-99ec-9304cd39778c" (UID: "191ad0c4-d290-448b-99ec-9304cd39778c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:42:52.087261 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.087238 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:52.087361 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.087268 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:52.087361 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.087287 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:52.087560 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.087535 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-workload-socket" pod "191ad0c4-d290-448b-99ec-9304cd39778c" (UID: "191ad0c4-d290-448b-99ec-9304cd39778c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:42:52.088022 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.087996 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191ad0c4-d290-448b-99ec-9304cd39778c-istiod-ca-cert" pod "191ad0c4-d290-448b-99ec-9304cd39778c" (UID: "191ad0c4-d290-448b-99ec-9304cd39778c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:42:52.089269 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.089233 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-istio-envoy" pod "191ad0c4-d290-448b-99ec-9304cd39778c" (UID: "191ad0c4-d290-448b-99ec-9304cd39778c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:42:52.089419 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.089395 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/191ad0c4-d290-448b-99ec-9304cd39778c-istio-podinfo" pod "191ad0c4-d290-448b-99ec-9304cd39778c" (UID: "191ad0c4-d290-448b-99ec-9304cd39778c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:42:52.089524 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.089416 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191ad0c4-d290-448b-99ec-9304cd39778c-kube-api-access-t86tn" pod "191ad0c4-d290-448b-99ec-9304cd39778c" (UID: "191ad0c4-d290-448b-99ec-9304cd39778c"). InnerVolumeSpecName "kube-api-access-t86tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:42:52.089598 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.089539 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191ad0c4-d290-448b-99ec-9304cd39778c-istio-token" pod "191ad0c4-d290-448b-99ec-9304cd39778c" (UID: "191ad0c4-d290-448b-99ec-9304cd39778c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:42:52.189036 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.188940 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/191ad0c4-d290-448b-99ec-9304cd39778c-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:52.189036 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.188975 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t86tn\" (UniqueName: \"kubernetes.io/projected/191ad0c4-d290-448b-99ec-9304cd39778c-kube-api-access-t86tn\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:52.189036 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.188998 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191ad0c4-d290-448b-99ec-9304cd39778c-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:52.189036 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.189007 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:52.189036 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.189016 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191ad0c4-d290-448b-99ec-9304cd39778c-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:52.189036 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.189024 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191ad0c4-d290-448b-99ec-9304cd39778c-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:52.358930 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.358888 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr"] Nov 02 23:42:52.362775 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.362746 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-v7rbr"] Nov 02 23:42:52.793375 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:42:52.793331 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="191ad0c4-d290-448b-99ec-9304cd39778c" path="/var/lib/kubelet/pods/191ad0c4-d290-448b-99ec-9304cd39778c/volumes" Nov 02 23:43:07.583323 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:43:07.583278 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:43:23.466048 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:23.466008 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62647c61-bhpt4"] Nov 02 23:43:23.466653 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:23.466633 5114 cpu_manager.go:414] "containerMap: removing container" podUID="191ad0c4-d290-448b-99ec-9304cd39778c" containerName="istio-proxy" Nov 02 23:43:23.466736 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:23.466655 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="191ad0c4-d290-448b-99ec-9304cd39778c" containerName="istio-proxy" Nov 02 23:43:23.466793 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:23.466780 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="191ad0c4-d290-448b-99ec-9304cd39778c" containerName="istio-proxy" Nov 02 23:43:23.469679 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:23.469657 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62647c61-bhpt4" Nov 02 23:43:23.527281 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:23.527246 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62647c61-bhpt4"] Nov 02 23:43:23.568106 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:23.568069 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd51a61b-e410-4996-8e72-16bd6e6a7881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62647c61-bhpt4\" (UID: \"fd51a61b-e410-4996-8e72-16bd6e6a7881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62647c61-bhpt4" Nov 02 23:43:23.568305 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:23.568188 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qstk4\" (UniqueName: \"kubernetes.io/projected/fd51a61b-e410-4996-8e72-16bd6e6a7881-kube-api-access-qstk4\") pod \"maas-api-revoke-sub-62647c61-bhpt4\" (UID: \"fd51a61b-e410-4996-8e72-16bd6e6a7881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62647c61-bhpt4" Nov 02 23:43:23.568390 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:23.568361 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd51a61b-e410-4996-8e72-16bd6e6a7881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62647c61-bhpt4\" (UID: \"fd51a61b-e410-4996-8e72-16bd6e6a7881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62647c61-bhpt4" Nov 02 23:43:23.574090 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:23.574063 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:43:23.574220 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:23.574166 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:43:23.668981 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:23.668944 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd51a61b-e410-4996-8e72-16bd6e6a7881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62647c61-bhpt4\" (UID: \"fd51a61b-e410-4996-8e72-16bd6e6a7881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62647c61-bhpt4" Nov 02 23:43:23.669176 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:23.669011 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd51a61b-e410-4996-8e72-16bd6e6a7881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62647c61-bhpt4\" (UID: \"fd51a61b-e410-4996-8e72-16bd6e6a7881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62647c61-bhpt4" Nov 02 23:43:23.669176 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:23.669065 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qstk4\" (UniqueName: \"kubernetes.io/projected/fd51a61b-e410-4996-8e72-16bd6e6a7881-kube-api-access-qstk4\") pod \"maas-api-revoke-sub-62647c61-bhpt4\" (UID: \"fd51a61b-e410-4996-8e72-16bd6e6a7881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62647c61-bhpt4" Nov 02 23:43:23.669949 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:23.669917 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd51a61b-e410-4996-8e72-16bd6e6a7881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62647c61-bhpt4\" (UID: \"fd51a61b-e410-4996-8e72-16bd6e6a7881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62647c61-bhpt4" Nov 02 23:43:23.671708 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:23.671684 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd51a61b-e410-4996-8e72-16bd6e6a7881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62647c61-bhpt4\" (UID: \"fd51a61b-e410-4996-8e72-16bd6e6a7881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62647c61-bhpt4" Nov 02 23:43:23.730156 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:23.730087 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qstk4\" (UniqueName: \"kubernetes.io/projected/fd51a61b-e410-4996-8e72-16bd6e6a7881-kube-api-access-qstk4\") pod \"maas-api-revoke-sub-62647c61-bhpt4\" (UID: \"fd51a61b-e410-4996-8e72-16bd6e6a7881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62647c61-bhpt4" Nov 02 23:43:23.783323 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:23.783284 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62647c61-bhpt4" Nov 02 23:43:24.162166 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:24.162127 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62647c61-bhpt4"] Nov 02 23:43:24.165780 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:43:24.165747 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd51a61b_e410_4996_8e72_16bd6e6a7881.slice/crio-364f5b54a23f822e6373e53f09b81e12ff31deb437dc5590a3627c800cddc03b WatchSource:0}: Error finding container 364f5b54a23f822e6373e53f09b81e12ff31deb437dc5590a3627c800cddc03b: Status 404 returned error can't find the container with id 364f5b54a23f822e6373e53f09b81e12ff31deb437dc5590a3627c800cddc03b Nov 02 23:43:25.115980 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:25.115899 5114 generic.go:356] "Generic (PLEG): container finished" podID="fa21ce01-b481-4cd5-bbba-8e00e9f9769d" containerID="ddf7ba4f39f9b0ce871e7dc4047d4bcbaed49ae4eb612205e29296a9ce5deae0" exitCode=0 Nov 02 23:43:25.115980 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:25.115969 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" event={"ID":"fa21ce01-b481-4cd5-bbba-8e00e9f9769d","Type":"ContainerDied","Data":"ddf7ba4f39f9b0ce871e7dc4047d4bcbaed49ae4eb612205e29296a9ce5deae0"} Nov 02 23:43:25.116542 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:25.116409 5114 scope.go:122] "RemoveContainer" containerID="ddf7ba4f39f9b0ce871e7dc4047d4bcbaed49ae4eb612205e29296a9ce5deae0" Nov 02 23:43:25.118074 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:25.118043 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:43:25.118211 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:25.118120 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:43:25.118211 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:25.118158 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:43:25.118319 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:25.118226 5114 generic.go:356] "Generic (PLEG): container finished" podID="fd51a61b-e410-4996-8e72-16bd6e6a7881" containerID="8bf11aedd8636603f471383fe35af645c8022c13a6d1d7a5377100d1ad21e8e9" exitCode=0 Nov 02 23:43:25.118319 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:25.118279 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62647c61-bhpt4" event={"ID":"fd51a61b-e410-4996-8e72-16bd6e6a7881","Type":"ContainerDied","Data":"8bf11aedd8636603f471383fe35af645c8022c13a6d1d7a5377100d1ad21e8e9"} Nov 02 23:43:25.118423 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:25.118332 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62647c61-bhpt4" event={"ID":"fd51a61b-e410-4996-8e72-16bd6e6a7881","Type":"ContainerStarted","Data":"364f5b54a23f822e6373e53f09b81e12ff31deb437dc5590a3627c800cddc03b"} Nov 02 23:43:26.138659 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:26.138607 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" event={"ID":"fa21ce01-b481-4cd5-bbba-8e00e9f9769d","Type":"ContainerStarted","Data":"3db1cd58f0144e9c24ed5ac65cdfe0aa5511e0732e9a2fecf2564e31c6c2ee69"} Nov 02 23:43:26.296797 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:26.296767 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62647c61-bhpt4" Nov 02 23:43:26.398049 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:26.397968 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd51a61b-e410-4996-8e72-16bd6e6a7881-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd51a61b-e410-4996-8e72-16bd6e6a7881-maas-api-openshift-ca\") pod \"fd51a61b-e410-4996-8e72-16bd6e6a7881\" (UID: \"fd51a61b-e410-4996-8e72-16bd6e6a7881\") " Nov 02 23:43:26.398049 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:26.398011 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd51a61b-e410-4996-8e72-16bd6e6a7881-kube-api-access-qstk4\" (UniqueName: \"kubernetes.io/projected/fd51a61b-e410-4996-8e72-16bd6e6a7881-kube-api-access-qstk4\") pod \"fd51a61b-e410-4996-8e72-16bd6e6a7881\" (UID: \"fd51a61b-e410-4996-8e72-16bd6e6a7881\") " Nov 02 23:43:26.398243 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:26.398143 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd51a61b-e410-4996-8e72-16bd6e6a7881-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd51a61b-e410-4996-8e72-16bd6e6a7881-maas-api-xks-ca\") pod \"fd51a61b-e410-4996-8e72-16bd6e6a7881\" (UID: \"fd51a61b-e410-4996-8e72-16bd6e6a7881\") " Nov 02 23:43:26.398425 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:26.398398 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd51a61b-e410-4996-8e72-16bd6e6a7881-maas-api-openshift-ca" pod "fd51a61b-e410-4996-8e72-16bd6e6a7881" (UID: "fd51a61b-e410-4996-8e72-16bd6e6a7881"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:43:26.400364 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:26.400335 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd51a61b-e410-4996-8e72-16bd6e6a7881-kube-api-access-qstk4" pod "fd51a61b-e410-4996-8e72-16bd6e6a7881" (UID: "fd51a61b-e410-4996-8e72-16bd6e6a7881"). InnerVolumeSpecName "kube-api-access-qstk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:43:26.400520 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:26.400398 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd51a61b-e410-4996-8e72-16bd6e6a7881-maas-api-xks-ca" pod "fd51a61b-e410-4996-8e72-16bd6e6a7881" (UID: "fd51a61b-e410-4996-8e72-16bd6e6a7881"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:43:26.500065 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:26.500018 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd51a61b-e410-4996-8e72-16bd6e6a7881-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:43:26.500065 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:26.500057 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd51a61b-e410-4996-8e72-16bd6e6a7881-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:43:26.500065 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:26.500071 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qstk4\" (UniqueName: \"kubernetes.io/projected/fd51a61b-e410-4996-8e72-16bd6e6a7881-kube-api-access-qstk4\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:43:26.666951 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:26.666853 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:43:26.667680 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:26.667657 5114 patch_prober.go:29] interesting pod/e2e-autores-875049-openshift-default-574965c6f-jzzx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 23:43:26.667745 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:26.667712 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" podUID="fa21ce01-b481-4cd5-bbba-8e00e9f9769d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 23:43:27.145823 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:27.145791 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62647c61-bhpt4" Nov 02 23:43:27.146236 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:27.145825 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62647c61-bhpt4" event={"ID":"fd51a61b-e410-4996-8e72-16bd6e6a7881","Type":"ContainerDied","Data":"364f5b54a23f822e6373e53f09b81e12ff31deb437dc5590a3627c800cddc03b"} Nov 02 23:43:27.146236 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:27.145877 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="364f5b54a23f822e6373e53f09b81e12ff31deb437dc5590a3627c800cddc03b" Nov 02 23:43:27.667180 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:27.667147 5114 patch_prober.go:29] interesting pod/e2e-autores-875049-openshift-default-574965c6f-jzzx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 23:43:27.667409 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:27.667213 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" podUID="fa21ce01-b481-4cd5-bbba-8e00e9f9769d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 23:43:28.667285 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:28.667247 5114 patch_prober.go:29] interesting pod/e2e-autores-875049-openshift-default-574965c6f-jzzx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 23:43:28.667764 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:28.667307 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" podUID="fa21ce01-b481-4cd5-bbba-8e00e9f9769d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 23:43:29.667048 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:29.667008 5114 patch_prober.go:29] interesting pod/e2e-autores-875049-openshift-default-574965c6f-jzzx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 23:43:29.667247 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:29.667079 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" podUID="fa21ce01-b481-4cd5-bbba-8e00e9f9769d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 23:43:30.667107 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:30.667065 5114 patch_prober.go:29] interesting pod/e2e-autores-875049-openshift-default-574965c6f-jzzx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 23:43:30.667582 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:30.667138 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" podUID="fa21ce01-b481-4cd5-bbba-8e00e9f9769d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 23:43:31.671991 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:31.671950 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:43:31.672537 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:31.672273 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:43:31.673170 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:31.673152 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:43:34.198955 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:34.198922 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebe2f460-mlr4n"] Nov 02 23:43:34.199381 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:34.199366 5114 cpu_manager.go:414] "containerMap: removing container" podUID="fd51a61b-e410-4996-8e72-16bd6e6a7881" containerName="revoke-keys" Nov 02 23:43:34.199645 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:34.199383 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd51a61b-e410-4996-8e72-16bd6e6a7881" containerName="revoke-keys" Nov 02 23:43:34.199645 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:34.199512 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd51a61b-e410-4996-8e72-16bd6e6a7881" containerName="revoke-keys" Nov 02 23:43:34.202307 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:34.202290 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe2f460-mlr4n" Nov 02 23:43:34.257368 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:34.257282 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:43:34.260318 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:34.260277 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebe2f460-mlr4n"] Nov 02 23:43:34.306940 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:34.306910 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:43:34.307396 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:34.307372 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9705213-5a0d-4ea0-b8b9-98af78097933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebe2f460-mlr4n\" (UID: \"b9705213-5a0d-4ea0-b8b9-98af78097933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe2f460-mlr4n" Nov 02 23:43:34.307520 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:34.307411 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv45c\" (UniqueName: \"kubernetes.io/projected/b9705213-5a0d-4ea0-b8b9-98af78097933-kube-api-access-tv45c\") pod \"maas-api-revoke-sub-ebe2f460-mlr4n\" (UID: \"b9705213-5a0d-4ea0-b8b9-98af78097933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe2f460-mlr4n" Nov 02 23:43:34.307605 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:34.307585 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9705213-5a0d-4ea0-b8b9-98af78097933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebe2f460-mlr4n\" (UID: \"b9705213-5a0d-4ea0-b8b9-98af78097933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe2f460-mlr4n" Nov 02 23:43:34.409222 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:34.409183 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9705213-5a0d-4ea0-b8b9-98af78097933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebe2f460-mlr4n\" (UID: \"b9705213-5a0d-4ea0-b8b9-98af78097933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe2f460-mlr4n" Nov 02 23:43:34.409415 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:34.409321 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9705213-5a0d-4ea0-b8b9-98af78097933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebe2f460-mlr4n\" (UID: \"b9705213-5a0d-4ea0-b8b9-98af78097933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe2f460-mlr4n" Nov 02 23:43:34.409415 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:34.409347 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv45c\" (UniqueName: \"kubernetes.io/projected/b9705213-5a0d-4ea0-b8b9-98af78097933-kube-api-access-tv45c\") pod \"maas-api-revoke-sub-ebe2f460-mlr4n\" (UID: \"b9705213-5a0d-4ea0-b8b9-98af78097933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe2f460-mlr4n" Nov 02 23:43:34.410086 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:34.410055 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9705213-5a0d-4ea0-b8b9-98af78097933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebe2f460-mlr4n\" (UID: \"b9705213-5a0d-4ea0-b8b9-98af78097933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe2f460-mlr4n" Nov 02 23:43:34.411756 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:34.411735 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9705213-5a0d-4ea0-b8b9-98af78097933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebe2f460-mlr4n\" (UID: \"b9705213-5a0d-4ea0-b8b9-98af78097933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe2f460-mlr4n" Nov 02 23:43:34.470376 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:34.470307 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv45c\" (UniqueName: \"kubernetes.io/projected/b9705213-5a0d-4ea0-b8b9-98af78097933-kube-api-access-tv45c\") pod \"maas-api-revoke-sub-ebe2f460-mlr4n\" (UID: \"b9705213-5a0d-4ea0-b8b9-98af78097933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe2f460-mlr4n" Nov 02 23:43:34.516472 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:34.516427 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe2f460-mlr4n" Nov 02 23:43:34.900967 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:34.900939 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebe2f460-mlr4n"] Nov 02 23:43:34.903265 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:43:34.903223 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9705213_5a0d_4ea0_b8b9_98af78097933.slice/crio-44db7fdc103b08eb28195567fcdef8d33ac991f18b17ea8b02860ecf73a5966b WatchSource:0}: Error finding container 44db7fdc103b08eb28195567fcdef8d33ac991f18b17ea8b02860ecf73a5966b: Status 404 returned error can't find the container with id 44db7fdc103b08eb28195567fcdef8d33ac991f18b17ea8b02860ecf73a5966b Nov 02 23:43:35.189203 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:35.189107 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe2f460-mlr4n" event={"ID":"b9705213-5a0d-4ea0-b8b9-98af78097933","Type":"ContainerStarted","Data":"44db7fdc103b08eb28195567fcdef8d33ac991f18b17ea8b02860ecf73a5966b"} Nov 02 23:43:36.196974 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:36.196885 5114 generic.go:356] "Generic (PLEG): container finished" podID="b9705213-5a0d-4ea0-b8b9-98af78097933" containerID="a2c8210681df33031f5bc9a35c4abbc43c98b71f7c1ad76e9affd4974604cd5d" exitCode=0 Nov 02 23:43:36.197357 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:36.196975 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe2f460-mlr4n" event={"ID":"b9705213-5a0d-4ea0-b8b9-98af78097933","Type":"ContainerDied","Data":"a2c8210681df33031f5bc9a35c4abbc43c98b71f7c1ad76e9affd4974604cd5d"} Nov 02 23:43:37.357769 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:37.357738 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe2f460-mlr4n" Nov 02 23:43:37.444631 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:37.444595 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9705213-5a0d-4ea0-b8b9-98af78097933-kube-api-access-tv45c\" (UniqueName: \"kubernetes.io/projected/b9705213-5a0d-4ea0-b8b9-98af78097933-kube-api-access-tv45c\") pod \"b9705213-5a0d-4ea0-b8b9-98af78097933\" (UID: \"b9705213-5a0d-4ea0-b8b9-98af78097933\") " Nov 02 23:43:37.444840 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:37.444654 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9705213-5a0d-4ea0-b8b9-98af78097933-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9705213-5a0d-4ea0-b8b9-98af78097933-maas-api-openshift-ca\") pod \"b9705213-5a0d-4ea0-b8b9-98af78097933\" (UID: \"b9705213-5a0d-4ea0-b8b9-98af78097933\") " Nov 02 23:43:37.444840 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:37.444696 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9705213-5a0d-4ea0-b8b9-98af78097933-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9705213-5a0d-4ea0-b8b9-98af78097933-maas-api-xks-ca\") pod \"b9705213-5a0d-4ea0-b8b9-98af78097933\" (UID: \"b9705213-5a0d-4ea0-b8b9-98af78097933\") " Nov 02 23:43:37.445019 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:37.444995 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9705213-5a0d-4ea0-b8b9-98af78097933-maas-api-openshift-ca" pod "b9705213-5a0d-4ea0-b8b9-98af78097933" (UID: "b9705213-5a0d-4ea0-b8b9-98af78097933"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:43:37.445203 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:37.445186 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9705213-5a0d-4ea0-b8b9-98af78097933-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:43:37.447006 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:37.446976 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9705213-5a0d-4ea0-b8b9-98af78097933-maas-api-xks-ca" pod "b9705213-5a0d-4ea0-b8b9-98af78097933" (UID: "b9705213-5a0d-4ea0-b8b9-98af78097933"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:43:37.447158 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:37.447020 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9705213-5a0d-4ea0-b8b9-98af78097933-kube-api-access-tv45c" pod "b9705213-5a0d-4ea0-b8b9-98af78097933" (UID: "b9705213-5a0d-4ea0-b8b9-98af78097933"). InnerVolumeSpecName "kube-api-access-tv45c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:43:37.546299 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:37.546254 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9705213-5a0d-4ea0-b8b9-98af78097933-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:43:37.546299 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:37.546289 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv45c\" (UniqueName: \"kubernetes.io/projected/b9705213-5a0d-4ea0-b8b9-98af78097933-kube-api-access-tv45c\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:43:37.585629 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:43:37.585592 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:43:38.209228 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:38.209196 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe2f460-mlr4n" Nov 02 23:43:38.209228 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:38.209198 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe2f460-mlr4n" event={"ID":"b9705213-5a0d-4ea0-b8b9-98af78097933","Type":"ContainerDied","Data":"44db7fdc103b08eb28195567fcdef8d33ac991f18b17ea8b02860ecf73a5966b"} Nov 02 23:43:38.209525 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:38.209294 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44db7fdc103b08eb28195567fcdef8d33ac991f18b17ea8b02860ecf73a5966b" Nov 02 23:43:44.454679 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.454640 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h"] Nov 02 23:43:44.455104 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.455033 5114 cpu_manager.go:414] "containerMap: removing container" podUID="b9705213-5a0d-4ea0-b8b9-98af78097933" containerName="revoke-keys" Nov 02 23:43:44.455104 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.455045 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9705213-5a0d-4ea0-b8b9-98af78097933" containerName="revoke-keys" Nov 02 23:43:44.455176 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.455141 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9705213-5a0d-4ea0-b8b9-98af78097933" containerName="revoke-keys" Nov 02 23:43:44.458321 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.458295 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:43:44.531333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.531287 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-model-cache\") pod \"autores-model-875049-kserve-5f4c5f78c7-rgx5h\" (UID: \"356fc994-5116-4aa2-9d6f-159c641f6fcd\") " pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:43:44.531541 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.531518 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-tmp-dir\") pod \"autores-model-875049-kserve-5f4c5f78c7-rgx5h\" (UID: \"356fc994-5116-4aa2-9d6f-159c641f6fcd\") " pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:43:44.531635 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.531605 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctz9v\" (UniqueName: \"kubernetes.io/projected/356fc994-5116-4aa2-9d6f-159c641f6fcd-kube-api-access-ctz9v\") pod \"autores-model-875049-kserve-5f4c5f78c7-rgx5h\" (UID: \"356fc994-5116-4aa2-9d6f-159c641f6fcd\") " pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:43:44.531756 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.531732 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-home\") pod \"autores-model-875049-kserve-5f4c5f78c7-rgx5h\" (UID: \"356fc994-5116-4aa2-9d6f-159c641f6fcd\") " pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:43:44.531861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.531837 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-dshm\") pod \"autores-model-875049-kserve-5f4c5f78c7-rgx5h\" (UID: \"356fc994-5116-4aa2-9d6f-159c641f6fcd\") " pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:43:44.531922 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.531862 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/356fc994-5116-4aa2-9d6f-159c641f6fcd-tls-certs\") pod \"autores-model-875049-kserve-5f4c5f78c7-rgx5h\" (UID: \"356fc994-5116-4aa2-9d6f-159c641f6fcd\") " pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:43:44.559794 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.559753 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-875049\"/\"kube-root-ca.crt\"" Nov 02 23:43:44.559794 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.559757 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-875049\"/\"default-dockercfg-xcjfw\"" Nov 02 23:43:44.560057 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.559816 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-875049\"/\"autores-model-875049-kserve-self-signed-certs\"" Nov 02 23:43:44.560057 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.559876 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-875049\"/\"openshift-service-ca.crt\"" Nov 02 23:43:44.562220 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.562197 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h"] Nov 02 23:43:44.633112 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.633067 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-dshm\") pod \"autores-model-875049-kserve-5f4c5f78c7-rgx5h\" (UID: \"356fc994-5116-4aa2-9d6f-159c641f6fcd\") " pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:43:44.633396 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.633376 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/356fc994-5116-4aa2-9d6f-159c641f6fcd-tls-certs\") pod \"autores-model-875049-kserve-5f4c5f78c7-rgx5h\" (UID: \"356fc994-5116-4aa2-9d6f-159c641f6fcd\") " pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:43:44.633572 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.633550 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-model-cache\") pod \"autores-model-875049-kserve-5f4c5f78c7-rgx5h\" (UID: \"356fc994-5116-4aa2-9d6f-159c641f6fcd\") " pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:43:44.633800 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.633785 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-tmp-dir\") pod \"autores-model-875049-kserve-5f4c5f78c7-rgx5h\" (UID: \"356fc994-5116-4aa2-9d6f-159c641f6fcd\") " pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:43:44.633920 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.633906 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctz9v\" (UniqueName: \"kubernetes.io/projected/356fc994-5116-4aa2-9d6f-159c641f6fcd-kube-api-access-ctz9v\") pod \"autores-model-875049-kserve-5f4c5f78c7-rgx5h\" (UID: \"356fc994-5116-4aa2-9d6f-159c641f6fcd\") " pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:43:44.634076 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.634064 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-home\") pod \"autores-model-875049-kserve-5f4c5f78c7-rgx5h\" (UID: \"356fc994-5116-4aa2-9d6f-159c641f6fcd\") " pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:43:44.634310 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.634283 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-model-cache\") pod \"autores-model-875049-kserve-5f4c5f78c7-rgx5h\" (UID: \"356fc994-5116-4aa2-9d6f-159c641f6fcd\") " pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:43:44.634613 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.634587 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-tmp-dir\") pod \"autores-model-875049-kserve-5f4c5f78c7-rgx5h\" (UID: \"356fc994-5116-4aa2-9d6f-159c641f6fcd\") " pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:43:44.634782 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.634753 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-home\") pod \"autores-model-875049-kserve-5f4c5f78c7-rgx5h\" (UID: \"356fc994-5116-4aa2-9d6f-159c641f6fcd\") " pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:43:44.636519 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.636470 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-dshm\") pod \"autores-model-875049-kserve-5f4c5f78c7-rgx5h\" (UID: \"356fc994-5116-4aa2-9d6f-159c641f6fcd\") " pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:43:44.637460 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.637434 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/356fc994-5116-4aa2-9d6f-159c641f6fcd-tls-certs\") pod \"autores-model-875049-kserve-5f4c5f78c7-rgx5h\" (UID: \"356fc994-5116-4aa2-9d6f-159c641f6fcd\") " pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:43:44.695817 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.695782 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctz9v\" (UniqueName: \"kubernetes.io/projected/356fc994-5116-4aa2-9d6f-159c641f6fcd-kube-api-access-ctz9v\") pod \"autores-model-875049-kserve-5f4c5f78c7-rgx5h\" (UID: \"356fc994-5116-4aa2-9d6f-159c641f6fcd\") " pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:43:44.821194 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.821156 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-875049\"/\"default-dockercfg-xcjfw\"" Nov 02 23:43:44.831296 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:44.831255 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:43:45.227622 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:45.227593 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h"] Nov 02 23:43:45.229464 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:43:45.229435 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod356fc994_5116_4aa2_9d6f_159c641f6fcd.slice/crio-2668d7bc1dab6e5089b4756fae4d1b2435a772d16b7e05998d8e4cc379f2ce73 WatchSource:0}: Error finding container 2668d7bc1dab6e5089b4756fae4d1b2435a772d16b7e05998d8e4cc379f2ce73: Status 404 returned error can't find the container with id 2668d7bc1dab6e5089b4756fae4d1b2435a772d16b7e05998d8e4cc379f2ce73 Nov 02 23:43:45.253998 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:45.253959 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" event={"ID":"356fc994-5116-4aa2-9d6f-159c641f6fcd","Type":"ContainerStarted","Data":"2668d7bc1dab6e5089b4756fae4d1b2435a772d16b7e05998d8e4cc379f2ce73"} Nov 02 23:43:46.260374 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:46.260329 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" event={"ID":"356fc994-5116-4aa2-9d6f-159c641f6fcd","Type":"ContainerStarted","Data":"216fa9ca31ab0517bcd5910e6238cf25898a8b60fffd1e2f5b7ee7d1efdabdc7"} Nov 02 23:43:46.377083 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:46.377027 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" podStartSLOduration=2.377010682 podStartE2EDuration="2.377010682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:43:46.375965824 +0000 UTC m=+3301.728999199" watchObservedRunningTime="2026-11-02 23:43:46.377010682 +0000 UTC m=+3301.730044342" Nov 02 23:43:46.832102 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:46.832061 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:43:54.831886 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:54.831846 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:43:54.844871 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:54.844838 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:43:55.329243 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:43:55.329210 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:44:03.493858 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:03.493822 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x"] Nov 02 23:44:07.584876 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:44:07.584835 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:44:11.315469 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:11.315430 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f4298b7-sqdqg"] Nov 02 23:44:11.320387 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:11.320369 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4298b7-sqdqg" Nov 02 23:44:11.422987 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:11.422930 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:44:11.422987 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:11.422947 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:44:11.425240 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:11.425220 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f4298b7-sqdqg"] Nov 02 23:44:11.449081 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:11.449041 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0330f53-39ae-458a-8a80-e80734e2c768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f4298b7-sqdqg\" (UID: \"e0330f53-39ae-458a-8a80-e80734e2c768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4298b7-sqdqg" Nov 02 23:44:11.449288 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:11.449198 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9cj8\" (UniqueName: \"kubernetes.io/projected/e0330f53-39ae-458a-8a80-e80734e2c768-kube-api-access-t9cj8\") pod \"maas-api-revoke-sub-9f4298b7-sqdqg\" (UID: \"e0330f53-39ae-458a-8a80-e80734e2c768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4298b7-sqdqg" Nov 02 23:44:11.449288 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:11.449275 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0330f53-39ae-458a-8a80-e80734e2c768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f4298b7-sqdqg\" (UID: \"e0330f53-39ae-458a-8a80-e80734e2c768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4298b7-sqdqg" Nov 02 23:44:11.550026 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:11.549986 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0330f53-39ae-458a-8a80-e80734e2c768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f4298b7-sqdqg\" (UID: \"e0330f53-39ae-458a-8a80-e80734e2c768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4298b7-sqdqg" Nov 02 23:44:11.550199 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:11.550104 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0330f53-39ae-458a-8a80-e80734e2c768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f4298b7-sqdqg\" (UID: \"e0330f53-39ae-458a-8a80-e80734e2c768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4298b7-sqdqg" Nov 02 23:44:11.550248 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:11.550205 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9cj8\" (UniqueName: \"kubernetes.io/projected/e0330f53-39ae-458a-8a80-e80734e2c768-kube-api-access-t9cj8\") pod \"maas-api-revoke-sub-9f4298b7-sqdqg\" (UID: \"e0330f53-39ae-458a-8a80-e80734e2c768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4298b7-sqdqg" Nov 02 23:44:11.550837 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:11.550804 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0330f53-39ae-458a-8a80-e80734e2c768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f4298b7-sqdqg\" (UID: \"e0330f53-39ae-458a-8a80-e80734e2c768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4298b7-sqdqg" Nov 02 23:44:11.552413 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:11.552390 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0330f53-39ae-458a-8a80-e80734e2c768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f4298b7-sqdqg\" (UID: \"e0330f53-39ae-458a-8a80-e80734e2c768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4298b7-sqdqg" Nov 02 23:44:11.611842 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:11.611774 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9cj8\" (UniqueName: \"kubernetes.io/projected/e0330f53-39ae-458a-8a80-e80734e2c768-kube-api-access-t9cj8\") pod \"maas-api-revoke-sub-9f4298b7-sqdqg\" (UID: \"e0330f53-39ae-458a-8a80-e80734e2c768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4298b7-sqdqg" Nov 02 23:44:11.634997 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:11.634958 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4298b7-sqdqg" Nov 02 23:44:12.008455 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:12.008426 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f4298b7-sqdqg"] Nov 02 23:44:12.010117 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:44:12.010083 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0330f53_39ae_458a_8a80_e80734e2c768.slice/crio-ecf5c88097db87027e637d7632e40c8baeaf69ba4490f229e5caf4a788004c6d WatchSource:0}: Error finding container ecf5c88097db87027e637d7632e40c8baeaf69ba4490f229e5caf4a788004c6d: Status 404 returned error can't find the container with id ecf5c88097db87027e637d7632e40c8baeaf69ba4490f229e5caf4a788004c6d Nov 02 23:44:12.415871 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:12.415835 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4298b7-sqdqg" event={"ID":"e0330f53-39ae-458a-8a80-e80734e2c768","Type":"ContainerStarted","Data":"bdac3c3fd7c7bc7d821d524aeba070eb623fd4f88ca3c67c347e99e6551fb4dd"} Nov 02 23:44:12.416296 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:12.415892 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4298b7-sqdqg" event={"ID":"e0330f53-39ae-458a-8a80-e80734e2c768","Type":"ContainerStarted","Data":"ecf5c88097db87027e637d7632e40c8baeaf69ba4490f229e5caf4a788004c6d"} Nov 02 23:44:12.530996 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:12.530932 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4298b7-sqdqg" podStartSLOduration=1.5309159079999999 podStartE2EDuration="1.530915908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:44:12.530298608 +0000 UTC m=+3327.883331986" watchObservedRunningTime="2026-11-02 23:44:12.530915908 +0000 UTC m=+3327.883949308" Nov 02 23:44:13.423932 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:13.423897 5114 generic.go:356] "Generic (PLEG): container finished" podID="e0330f53-39ae-458a-8a80-e80734e2c768" containerID="bdac3c3fd7c7bc7d821d524aeba070eb623fd4f88ca3c67c347e99e6551fb4dd" exitCode=0 Nov 02 23:44:13.424369 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:13.423983 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4298b7-sqdqg" event={"ID":"e0330f53-39ae-458a-8a80-e80734e2c768","Type":"ContainerDied","Data":"bdac3c3fd7c7bc7d821d524aeba070eb623fd4f88ca3c67c347e99e6551fb4dd"} Nov 02 23:44:14.577120 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:14.577085 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4298b7-sqdqg" Nov 02 23:44:14.688772 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:14.688683 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0330f53-39ae-458a-8a80-e80734e2c768-kube-api-access-t9cj8\" (UniqueName: \"kubernetes.io/projected/e0330f53-39ae-458a-8a80-e80734e2c768-kube-api-access-t9cj8\") pod \"e0330f53-39ae-458a-8a80-e80734e2c768\" (UID: \"e0330f53-39ae-458a-8a80-e80734e2c768\") " Nov 02 23:44:14.688772 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:14.688740 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0330f53-39ae-458a-8a80-e80734e2c768-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0330f53-39ae-458a-8a80-e80734e2c768-maas-api-openshift-ca\") pod \"e0330f53-39ae-458a-8a80-e80734e2c768\" (UID: \"e0330f53-39ae-458a-8a80-e80734e2c768\") " Nov 02 23:44:14.688982 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:14.688824 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0330f53-39ae-458a-8a80-e80734e2c768-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0330f53-39ae-458a-8a80-e80734e2c768-maas-api-xks-ca\") pod \"e0330f53-39ae-458a-8a80-e80734e2c768\" (UID: \"e0330f53-39ae-458a-8a80-e80734e2c768\") " Nov 02 23:44:14.689257 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:14.689232 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0330f53-39ae-458a-8a80-e80734e2c768-maas-api-openshift-ca" pod "e0330f53-39ae-458a-8a80-e80734e2c768" (UID: "e0330f53-39ae-458a-8a80-e80734e2c768"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:44:14.691093 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:14.691044 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0330f53-39ae-458a-8a80-e80734e2c768-kube-api-access-t9cj8" pod "e0330f53-39ae-458a-8a80-e80734e2c768" (UID: "e0330f53-39ae-458a-8a80-e80734e2c768"). InnerVolumeSpecName "kube-api-access-t9cj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:44:14.691093 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:14.691045 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0330f53-39ae-458a-8a80-e80734e2c768-maas-api-xks-ca" pod "e0330f53-39ae-458a-8a80-e80734e2c768" (UID: "e0330f53-39ae-458a-8a80-e80734e2c768"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:44:14.790773 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:14.790743 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9cj8\" (UniqueName: \"kubernetes.io/projected/e0330f53-39ae-458a-8a80-e80734e2c768-kube-api-access-t9cj8\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:44:14.790773 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:14.790770 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0330f53-39ae-458a-8a80-e80734e2c768-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:44:14.790773 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:14.790780 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0330f53-39ae-458a-8a80-e80734e2c768-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:44:15.436008 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:15.435976 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4298b7-sqdqg" Nov 02 23:44:15.436008 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:15.435994 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4298b7-sqdqg" event={"ID":"e0330f53-39ae-458a-8a80-e80734e2c768","Type":"ContainerDied","Data":"ecf5c88097db87027e637d7632e40c8baeaf69ba4490f229e5caf4a788004c6d"} Nov 02 23:44:15.436245 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:15.436046 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecf5c88097db87027e637d7632e40c8baeaf69ba4490f229e5caf4a788004c6d" Nov 02 23:44:18.514651 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:18.514599 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" podUID="7a521ccd-9afc-4dcb-af5e-de54dabb7340" containerName="main" containerID="cri-o://0c5963ce5657d78689de0e67a689f339ace1270e6cddf05ffb36b4413055663f" gracePeriod=45 Nov 02 23:44:18.779568 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:18.779459 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:44:18.833170 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:18.833137 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-home\" (UniqueName: \"kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-home\") pod \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\" (UID: \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\") " Nov 02 23:44:18.833350 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:18.833226 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-model-cache\") pod \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\" (UID: \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\") " Nov 02 23:44:18.833350 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:18.833275 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a521ccd-9afc-4dcb-af5e-de54dabb7340-kube-api-access-9qjbr\" (UniqueName: \"kubernetes.io/projected/7a521ccd-9afc-4dcb-af5e-de54dabb7340-kube-api-access-9qjbr\") pod \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\" (UID: \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\") " Nov 02 23:44:18.833350 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:18.833313 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a521ccd-9afc-4dcb-af5e-de54dabb7340-tls-certs\" (UniqueName: \"kubernetes.io/secret/7a521ccd-9afc-4dcb-af5e-de54dabb7340-tls-certs\") pod \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\" (UID: \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\") " Nov 02 23:44:18.833538 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:18.833374 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-tmp-dir\") pod \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\" (UID: \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\") " Nov 02 23:44:18.833538 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:18.833428 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-home" pod "7a521ccd-9afc-4dcb-af5e-de54dabb7340" (UID: "7a521ccd-9afc-4dcb-af5e-de54dabb7340"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:44:18.833538 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:18.833449 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-dshm\") pod \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\" (UID: \"7a521ccd-9afc-4dcb-af5e-de54dabb7340\") " Nov 02 23:44:18.833538 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:18.833460 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-model-cache" pod "7a521ccd-9afc-4dcb-af5e-de54dabb7340" (UID: "7a521ccd-9afc-4dcb-af5e-de54dabb7340"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:44:18.833758 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:18.833698 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-tmp-dir" pod "7a521ccd-9afc-4dcb-af5e-de54dabb7340" (UID: "7a521ccd-9afc-4dcb-af5e-de54dabb7340"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:44:18.833927 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:18.833906 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:44:18.833927 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:18.833929 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:44:18.834096 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:18.833943 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:44:18.835917 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:18.835888 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a521ccd-9afc-4dcb-af5e-de54dabb7340-tls-certs" pod "7a521ccd-9afc-4dcb-af5e-de54dabb7340" (UID: "7a521ccd-9afc-4dcb-af5e-de54dabb7340"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:44:18.835917 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:18.835898 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-dshm" pod "7a521ccd-9afc-4dcb-af5e-de54dabb7340" (UID: "7a521ccd-9afc-4dcb-af5e-de54dabb7340"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:44:18.836058 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:18.835962 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a521ccd-9afc-4dcb-af5e-de54dabb7340-kube-api-access-9qjbr" pod "7a521ccd-9afc-4dcb-af5e-de54dabb7340" (UID: "7a521ccd-9afc-4dcb-af5e-de54dabb7340"). InnerVolumeSpecName "kube-api-access-9qjbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:44:18.934842 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:18.934798 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a521ccd-9afc-4dcb-af5e-de54dabb7340-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:44:18.934842 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:18.934833 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qjbr\" (UniqueName: \"kubernetes.io/projected/7a521ccd-9afc-4dcb-af5e-de54dabb7340-kube-api-access-9qjbr\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:44:18.934842 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:18.934845 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a521ccd-9afc-4dcb-af5e-de54dabb7340-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:44:19.462906 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:19.462866 5114 generic.go:356] "Generic (PLEG): container finished" podID="7a521ccd-9afc-4dcb-af5e-de54dabb7340" containerID="0c5963ce5657d78689de0e67a689f339ace1270e6cddf05ffb36b4413055663f" exitCode=0 Nov 02 23:44:19.463124 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:19.462953 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" Nov 02 23:44:19.463124 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:19.462954 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" event={"ID":"7a521ccd-9afc-4dcb-af5e-de54dabb7340","Type":"ContainerDied","Data":"0c5963ce5657d78689de0e67a689f339ace1270e6cddf05ffb36b4413055663f"} Nov 02 23:44:19.463124 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:19.463073 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x" event={"ID":"7a521ccd-9afc-4dcb-af5e-de54dabb7340","Type":"ContainerDied","Data":"b9b79e46bcbaf512346ac648356b29b0077aa1da6a886aa4e803dc639dc350e6"} Nov 02 23:44:19.463124 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:19.463094 5114 scope.go:122] "RemoveContainer" containerID="0c5963ce5657d78689de0e67a689f339ace1270e6cddf05ffb36b4413055663f" Nov 02 23:44:19.473953 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:19.473927 5114 scope.go:122] "RemoveContainer" containerID="0c5963ce5657d78689de0e67a689f339ace1270e6cddf05ffb36b4413055663f" Nov 02 23:44:19.474360 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:44:19.474340 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c5963ce5657d78689de0e67a689f339ace1270e6cddf05ffb36b4413055663f\": container with ID starting with 0c5963ce5657d78689de0e67a689f339ace1270e6cddf05ffb36b4413055663f not found: ID does not exist" containerID="0c5963ce5657d78689de0e67a689f339ace1270e6cddf05ffb36b4413055663f" Nov 02 23:44:19.474416 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:19.474371 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c5963ce5657d78689de0e67a689f339ace1270e6cddf05ffb36b4413055663f"} err="failed to get container status \"0c5963ce5657d78689de0e67a689f339ace1270e6cddf05ffb36b4413055663f\": rpc error: code = NotFound desc = could not find container \"0c5963ce5657d78689de0e67a689f339ace1270e6cddf05ffb36b4413055663f\": container with ID starting with 0c5963ce5657d78689de0e67a689f339ace1270e6cddf05ffb36b4413055663f not found: ID does not exist" Nov 02 23:44:19.625433 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:19.625393 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x"] Nov 02 23:44:19.675260 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:19.675223 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f46b61/auth-test-model-f46b61-kserve-c69cf7585-vh62x"] Nov 02 23:44:20.794012 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:20.793977 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a521ccd-9afc-4dcb-af5e-de54dabb7340" path="/var/lib/kubelet/pods/7a521ccd-9afc-4dcb-af5e-de54dabb7340/volumes" Nov 02 23:44:22.063258 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:22.063209 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f30cfd3a-m7tmd"] Nov 02 23:44:22.063863 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:22.063841 5114 cpu_manager.go:414] "containerMap: removing container" podUID="7a521ccd-9afc-4dcb-af5e-de54dabb7340" containerName="main" Nov 02 23:44:22.063952 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:22.063865 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a521ccd-9afc-4dcb-af5e-de54dabb7340" containerName="main" Nov 02 23:44:22.063952 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:22.063897 5114 cpu_manager.go:414] "containerMap: removing container" podUID="e0330f53-39ae-458a-8a80-e80734e2c768" containerName="revoke-keys" Nov 02 23:44:22.063952 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:22.063905 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0330f53-39ae-458a-8a80-e80734e2c768" containerName="revoke-keys" Nov 02 23:44:22.064107 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:22.064031 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a521ccd-9afc-4dcb-af5e-de54dabb7340" containerName="main" Nov 02 23:44:22.064107 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:22.064049 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0330f53-39ae-458a-8a80-e80734e2c768" containerName="revoke-keys" Nov 02 23:44:22.068545 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:22.068520 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cfd3a-m7tmd" Nov 02 23:44:22.126287 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:22.126242 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f30cfd3a-m7tmd"] Nov 02 23:44:22.172121 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:22.172082 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-794dl\" (UniqueName: \"kubernetes.io/projected/00962ce7-ca9b-450b-b68d-bbb0165a7553-kube-api-access-794dl\") pod \"maas-api-revoke-sub-f30cfd3a-m7tmd\" (UID: \"00962ce7-ca9b-450b-b68d-bbb0165a7553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cfd3a-m7tmd" Nov 02 23:44:22.172336 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:22.172130 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00962ce7-ca9b-450b-b68d-bbb0165a7553-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f30cfd3a-m7tmd\" (UID: \"00962ce7-ca9b-450b-b68d-bbb0165a7553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cfd3a-m7tmd" Nov 02 23:44:22.172336 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:22.172235 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00962ce7-ca9b-450b-b68d-bbb0165a7553-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f30cfd3a-m7tmd\" (UID: \"00962ce7-ca9b-450b-b68d-bbb0165a7553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cfd3a-m7tmd" Nov 02 23:44:22.172437 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:22.172390 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:44:22.172437 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:22.172430 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:44:22.273438 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:22.273399 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-794dl\" (UniqueName: \"kubernetes.io/projected/00962ce7-ca9b-450b-b68d-bbb0165a7553-kube-api-access-794dl\") pod \"maas-api-revoke-sub-f30cfd3a-m7tmd\" (UID: \"00962ce7-ca9b-450b-b68d-bbb0165a7553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cfd3a-m7tmd" Nov 02 23:44:22.273438 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:22.273441 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00962ce7-ca9b-450b-b68d-bbb0165a7553-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f30cfd3a-m7tmd\" (UID: \"00962ce7-ca9b-450b-b68d-bbb0165a7553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cfd3a-m7tmd" Nov 02 23:44:22.273703 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:22.273466 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00962ce7-ca9b-450b-b68d-bbb0165a7553-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f30cfd3a-m7tmd\" (UID: \"00962ce7-ca9b-450b-b68d-bbb0165a7553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cfd3a-m7tmd" Nov 02 23:44:22.274119 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:22.274088 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00962ce7-ca9b-450b-b68d-bbb0165a7553-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f30cfd3a-m7tmd\" (UID: \"00962ce7-ca9b-450b-b68d-bbb0165a7553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cfd3a-m7tmd" Nov 02 23:44:22.276721 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:22.276702 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00962ce7-ca9b-450b-b68d-bbb0165a7553-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f30cfd3a-m7tmd\" (UID: \"00962ce7-ca9b-450b-b68d-bbb0165a7553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cfd3a-m7tmd" Nov 02 23:44:22.334427 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:22.334353 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-794dl\" (UniqueName: \"kubernetes.io/projected/00962ce7-ca9b-450b-b68d-bbb0165a7553-kube-api-access-794dl\") pod \"maas-api-revoke-sub-f30cfd3a-m7tmd\" (UID: \"00962ce7-ca9b-450b-b68d-bbb0165a7553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cfd3a-m7tmd" Nov 02 23:44:22.381853 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:22.381814 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cfd3a-m7tmd" Nov 02 23:44:22.759636 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:22.759608 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f30cfd3a-m7tmd"] Nov 02 23:44:22.762106 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:44:22.762080 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00962ce7_ca9b_450b_b68d_bbb0165a7553.slice/crio-703d3992728d33179ab1ab3feda4fec8a8a69ee460ab5f3a5751d59cc42aa507 WatchSource:0}: Error finding container 703d3992728d33179ab1ab3feda4fec8a8a69ee460ab5f3a5751d59cc42aa507: Status 404 returned error can't find the container with id 703d3992728d33179ab1ab3feda4fec8a8a69ee460ab5f3a5751d59cc42aa507 Nov 02 23:44:23.487242 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:23.487197 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cfd3a-m7tmd" event={"ID":"00962ce7-ca9b-450b-b68d-bbb0165a7553","Type":"ContainerStarted","Data":"19d3f6afbc1bb0c3b3aba2a4f0fa6e3b9f2aec194448f5fc5f62321711713280"} Nov 02 23:44:23.487739 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:23.487262 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cfd3a-m7tmd" event={"ID":"00962ce7-ca9b-450b-b68d-bbb0165a7553","Type":"ContainerStarted","Data":"703d3992728d33179ab1ab3feda4fec8a8a69ee460ab5f3a5751d59cc42aa507"} Nov 02 23:44:23.603757 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:23.603696 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cfd3a-m7tmd" podStartSLOduration=1.603676391 podStartE2EDuration="1.603676391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:44:23.601929611 +0000 UTC m=+3338.954962989" watchObservedRunningTime="2026-11-02 23:44:23.603676391 +0000 UTC m=+3338.956709767" Nov 02 23:44:24.494733 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:24.494638 5114 generic.go:356] "Generic (PLEG): container finished" podID="00962ce7-ca9b-450b-b68d-bbb0165a7553" containerID="19d3f6afbc1bb0c3b3aba2a4f0fa6e3b9f2aec194448f5fc5f62321711713280" exitCode=0 Nov 02 23:44:24.494733 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:24.494690 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cfd3a-m7tmd" event={"ID":"00962ce7-ca9b-450b-b68d-bbb0165a7553","Type":"ContainerDied","Data":"19d3f6afbc1bb0c3b3aba2a4f0fa6e3b9f2aec194448f5fc5f62321711713280"} Nov 02 23:44:25.652398 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:25.652368 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cfd3a-m7tmd" Nov 02 23:44:25.712609 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:25.712573 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00962ce7-ca9b-450b-b68d-bbb0165a7553-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00962ce7-ca9b-450b-b68d-bbb0165a7553-maas-api-openshift-ca\") pod \"00962ce7-ca9b-450b-b68d-bbb0165a7553\" (UID: \"00962ce7-ca9b-450b-b68d-bbb0165a7553\") " Nov 02 23:44:25.712796 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:25.712708 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00962ce7-ca9b-450b-b68d-bbb0165a7553-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00962ce7-ca9b-450b-b68d-bbb0165a7553-maas-api-xks-ca\") pod \"00962ce7-ca9b-450b-b68d-bbb0165a7553\" (UID: \"00962ce7-ca9b-450b-b68d-bbb0165a7553\") " Nov 02 23:44:25.712796 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:25.712739 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00962ce7-ca9b-450b-b68d-bbb0165a7553-kube-api-access-794dl\" (UniqueName: \"kubernetes.io/projected/00962ce7-ca9b-450b-b68d-bbb0165a7553-kube-api-access-794dl\") pod \"00962ce7-ca9b-450b-b68d-bbb0165a7553\" (UID: \"00962ce7-ca9b-450b-b68d-bbb0165a7553\") " Nov 02 23:44:25.713031 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:25.713002 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00962ce7-ca9b-450b-b68d-bbb0165a7553-maas-api-openshift-ca" pod "00962ce7-ca9b-450b-b68d-bbb0165a7553" (UID: "00962ce7-ca9b-450b-b68d-bbb0165a7553"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:44:25.714972 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:25.714940 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00962ce7-ca9b-450b-b68d-bbb0165a7553-maas-api-xks-ca" pod "00962ce7-ca9b-450b-b68d-bbb0165a7553" (UID: "00962ce7-ca9b-450b-b68d-bbb0165a7553"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:44:25.715107 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:25.714995 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00962ce7-ca9b-450b-b68d-bbb0165a7553-kube-api-access-794dl" pod "00962ce7-ca9b-450b-b68d-bbb0165a7553" (UID: "00962ce7-ca9b-450b-b68d-bbb0165a7553"). InnerVolumeSpecName "kube-api-access-794dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:44:25.814071 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:25.814032 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00962ce7-ca9b-450b-b68d-bbb0165a7553-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:44:25.814071 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:25.814067 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00962ce7-ca9b-450b-b68d-bbb0165a7553-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:44:25.814071 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:25.814078 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-794dl\" (UniqueName: \"kubernetes.io/projected/00962ce7-ca9b-450b-b68d-bbb0165a7553-kube-api-access-794dl\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:44:26.515290 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:26.515252 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cfd3a-m7tmd" Nov 02 23:44:26.515539 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:26.515297 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cfd3a-m7tmd" event={"ID":"00962ce7-ca9b-450b-b68d-bbb0165a7553","Type":"ContainerDied","Data":"703d3992728d33179ab1ab3feda4fec8a8a69ee460ab5f3a5751d59cc42aa507"} Nov 02 23:44:26.515539 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:26.515373 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="703d3992728d33179ab1ab3feda4fec8a8a69ee460ab5f3a5751d59cc42aa507" Nov 02 23:44:37.599442 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:44:37.599406 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:44:40.379937 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:40.379897 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f141ce6-dqg6b"] Nov 02 23:44:40.380369 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:40.380355 5114 cpu_manager.go:414] "containerMap: removing container" podUID="00962ce7-ca9b-450b-b68d-bbb0165a7553" containerName="revoke-keys" Nov 02 23:44:40.380416 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:40.380370 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00962ce7-ca9b-450b-b68d-bbb0165a7553" containerName="revoke-keys" Nov 02 23:44:40.380452 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:40.380437 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="00962ce7-ca9b-450b-b68d-bbb0165a7553" containerName="revoke-keys" Nov 02 23:44:40.384024 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:40.384007 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141ce6-dqg6b" Nov 02 23:44:40.441443 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:40.441410 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f141ce6-dqg6b"] Nov 02 23:44:40.453309 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:40.453269 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:44:40.453475 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:40.453405 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:44:40.483012 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:40.482972 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bea96b8-5ae1-425c-98de-c90603572fbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f141ce6-dqg6b\" (UID: \"1bea96b8-5ae1-425c-98de-c90603572fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141ce6-dqg6b" Nov 02 23:44:40.483223 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:40.483114 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-896hb\" (UniqueName: \"kubernetes.io/projected/1bea96b8-5ae1-425c-98de-c90603572fbc-kube-api-access-896hb\") pod \"maas-api-revoke-sub-6f141ce6-dqg6b\" (UID: \"1bea96b8-5ae1-425c-98de-c90603572fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141ce6-dqg6b" Nov 02 23:44:40.483280 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:40.483252 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bea96b8-5ae1-425c-98de-c90603572fbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f141ce6-dqg6b\" (UID: \"1bea96b8-5ae1-425c-98de-c90603572fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141ce6-dqg6b" Nov 02 23:44:40.584248 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:40.584209 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bea96b8-5ae1-425c-98de-c90603572fbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f141ce6-dqg6b\" (UID: \"1bea96b8-5ae1-425c-98de-c90603572fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141ce6-dqg6b" Nov 02 23:44:40.584454 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:40.584277 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-896hb\" (UniqueName: \"kubernetes.io/projected/1bea96b8-5ae1-425c-98de-c90603572fbc-kube-api-access-896hb\") pod \"maas-api-revoke-sub-6f141ce6-dqg6b\" (UID: \"1bea96b8-5ae1-425c-98de-c90603572fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141ce6-dqg6b" Nov 02 23:44:40.584454 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:40.584333 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bea96b8-5ae1-425c-98de-c90603572fbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f141ce6-dqg6b\" (UID: \"1bea96b8-5ae1-425c-98de-c90603572fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141ce6-dqg6b" Nov 02 23:44:40.584916 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:40.584895 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bea96b8-5ae1-425c-98de-c90603572fbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f141ce6-dqg6b\" (UID: \"1bea96b8-5ae1-425c-98de-c90603572fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141ce6-dqg6b" Nov 02 23:44:40.586863 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:40.586840 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bea96b8-5ae1-425c-98de-c90603572fbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f141ce6-dqg6b\" (UID: \"1bea96b8-5ae1-425c-98de-c90603572fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141ce6-dqg6b" Nov 02 23:44:40.645337 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:40.645253 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-896hb\" (UniqueName: \"kubernetes.io/projected/1bea96b8-5ae1-425c-98de-c90603572fbc-kube-api-access-896hb\") pod \"maas-api-revoke-sub-6f141ce6-dqg6b\" (UID: \"1bea96b8-5ae1-425c-98de-c90603572fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141ce6-dqg6b" Nov 02 23:44:40.697030 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:40.696989 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141ce6-dqg6b" Nov 02 23:44:41.081643 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:41.081615 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f141ce6-dqg6b"] Nov 02 23:44:41.084144 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:44:41.084119 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bea96b8_5ae1_425c_98de_c90603572fbc.slice/crio-19d780929ad32a7c2d0c9db1239c27673e0053fb27bd00da3d6c86291b73276f WatchSource:0}: Error finding container 19d780929ad32a7c2d0c9db1239c27673e0053fb27bd00da3d6c86291b73276f: Status 404 returned error can't find the container with id 19d780929ad32a7c2d0c9db1239c27673e0053fb27bd00da3d6c86291b73276f Nov 02 23:44:41.613715 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:41.613669 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141ce6-dqg6b" event={"ID":"1bea96b8-5ae1-425c-98de-c90603572fbc","Type":"ContainerStarted","Data":"3522547d8af74e06b25cbbe9b908758e1c59616933c8d0f6b9f82f2af3b385ad"} Nov 02 23:44:41.614141 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:41.613729 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141ce6-dqg6b" event={"ID":"1bea96b8-5ae1-425c-98de-c90603572fbc","Type":"ContainerStarted","Data":"19d780929ad32a7c2d0c9db1239c27673e0053fb27bd00da3d6c86291b73276f"} Nov 02 23:44:41.730184 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:41.730070 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141ce6-dqg6b" podStartSLOduration=1.73005252 podStartE2EDuration="1.73005252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:44:41.728331549 +0000 UTC m=+3357.081364924" watchObservedRunningTime="2026-11-02 23:44:41.73005252 +0000 UTC m=+3357.083085894" Nov 02 23:44:42.620387 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:42.620352 5114 generic.go:356] "Generic (PLEG): container finished" podID="1bea96b8-5ae1-425c-98de-c90603572fbc" containerID="3522547d8af74e06b25cbbe9b908758e1c59616933c8d0f6b9f82f2af3b385ad" exitCode=0 Nov 02 23:44:42.620834 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:42.620440 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141ce6-dqg6b" event={"ID":"1bea96b8-5ae1-425c-98de-c90603572fbc","Type":"ContainerDied","Data":"3522547d8af74e06b25cbbe9b908758e1c59616933c8d0f6b9f82f2af3b385ad"} Nov 02 23:44:43.772679 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:43.772651 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141ce6-dqg6b" Nov 02 23:44:43.921675 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:43.921569 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bea96b8-5ae1-425c-98de-c90603572fbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bea96b8-5ae1-425c-98de-c90603572fbc-maas-api-openshift-ca\") pod \"1bea96b8-5ae1-425c-98de-c90603572fbc\" (UID: \"1bea96b8-5ae1-425c-98de-c90603572fbc\") " Nov 02 23:44:43.921675 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:43.921635 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bea96b8-5ae1-425c-98de-c90603572fbc-kube-api-access-896hb\" (UniqueName: \"kubernetes.io/projected/1bea96b8-5ae1-425c-98de-c90603572fbc-kube-api-access-896hb\") pod \"1bea96b8-5ae1-425c-98de-c90603572fbc\" (UID: \"1bea96b8-5ae1-425c-98de-c90603572fbc\") " Nov 02 23:44:43.921937 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:43.921784 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bea96b8-5ae1-425c-98de-c90603572fbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bea96b8-5ae1-425c-98de-c90603572fbc-maas-api-xks-ca\") pod \"1bea96b8-5ae1-425c-98de-c90603572fbc\" (UID: \"1bea96b8-5ae1-425c-98de-c90603572fbc\") " Nov 02 23:44:43.921999 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:43.921962 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bea96b8-5ae1-425c-98de-c90603572fbc-maas-api-openshift-ca" pod "1bea96b8-5ae1-425c-98de-c90603572fbc" (UID: "1bea96b8-5ae1-425c-98de-c90603572fbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:44:43.922257 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:43.922237 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bea96b8-5ae1-425c-98de-c90603572fbc-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:44:43.924077 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:43.924038 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bea96b8-5ae1-425c-98de-c90603572fbc-maas-api-xks-ca" pod "1bea96b8-5ae1-425c-98de-c90603572fbc" (UID: "1bea96b8-5ae1-425c-98de-c90603572fbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:44:43.924236 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:43.924167 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bea96b8-5ae1-425c-98de-c90603572fbc-kube-api-access-896hb" pod "1bea96b8-5ae1-425c-98de-c90603572fbc" (UID: "1bea96b8-5ae1-425c-98de-c90603572fbc"). InnerVolumeSpecName "kube-api-access-896hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:44:44.023274 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:44.023225 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-896hb\" (UniqueName: \"kubernetes.io/projected/1bea96b8-5ae1-425c-98de-c90603572fbc-kube-api-access-896hb\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:44:44.023274 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:44.023266 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bea96b8-5ae1-425c-98de-c90603572fbc-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:44:44.633217 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:44.633181 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141ce6-dqg6b" Nov 02 23:44:44.633217 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:44.633194 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f141ce6-dqg6b" event={"ID":"1bea96b8-5ae1-425c-98de-c90603572fbc","Type":"ContainerDied","Data":"19d780929ad32a7c2d0c9db1239c27673e0053fb27bd00da3d6c86291b73276f"} Nov 02 23:44:44.633450 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:44.633283 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19d780929ad32a7c2d0c9db1239c27673e0053fb27bd00da3d6c86291b73276f" Nov 02 23:44:47.046975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:47.046940 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86a28697-7wkdc"] Nov 02 23:44:47.096374 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:47.096344 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86a28697-7wkdc"] Nov 02 23:44:48.794010 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:48.793971 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7d12b96-e660-4927-9424-b5ce36b78ac5" path="/var/lib/kubelet/pods/e7d12b96-e660-4927-9424-b5ce36b78ac5/volumes" Nov 02 23:44:51.119059 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:51.119022 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2d54581-266r9"] Nov 02 23:44:51.119496 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:51.119458 5114 cpu_manager.go:414] "containerMap: removing container" podUID="1bea96b8-5ae1-425c-98de-c90603572fbc" containerName="revoke-keys" Nov 02 23:44:51.119496 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:51.119471 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bea96b8-5ae1-425c-98de-c90603572fbc" containerName="revoke-keys" Nov 02 23:44:51.119592 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:51.119579 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bea96b8-5ae1-425c-98de-c90603572fbc" containerName="revoke-keys" Nov 02 23:44:51.122943 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:51.122922 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d54581-266r9" Nov 02 23:44:51.181527 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:51.181461 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2d54581-266r9"] Nov 02 23:44:51.204748 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:51.204709 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a54093d-f5a1-41af-b5db-deb571d294ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2d54581-266r9\" (UID: \"1a54093d-f5a1-41af-b5db-deb571d294ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d54581-266r9" Nov 02 23:44:51.204951 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:51.204756 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g78b4\" (UniqueName: \"kubernetes.io/projected/1a54093d-f5a1-41af-b5db-deb571d294ed-kube-api-access-g78b4\") pod \"maas-api-revoke-sub-a2d54581-266r9\" (UID: \"1a54093d-f5a1-41af-b5db-deb571d294ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d54581-266r9" Nov 02 23:44:51.204951 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:51.204917 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a54093d-f5a1-41af-b5db-deb571d294ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2d54581-266r9\" (UID: \"1a54093d-f5a1-41af-b5db-deb571d294ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d54581-266r9" Nov 02 23:44:51.226899 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:51.226866 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:44:51.227090 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:51.226919 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:44:51.306198 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:51.306156 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a54093d-f5a1-41af-b5db-deb571d294ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2d54581-266r9\" (UID: \"1a54093d-f5a1-41af-b5db-deb571d294ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d54581-266r9" Nov 02 23:44:51.306198 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:51.306200 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g78b4\" (UniqueName: \"kubernetes.io/projected/1a54093d-f5a1-41af-b5db-deb571d294ed-kube-api-access-g78b4\") pod \"maas-api-revoke-sub-a2d54581-266r9\" (UID: \"1a54093d-f5a1-41af-b5db-deb571d294ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d54581-266r9" Nov 02 23:44:51.306463 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:51.306298 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a54093d-f5a1-41af-b5db-deb571d294ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2d54581-266r9\" (UID: \"1a54093d-f5a1-41af-b5db-deb571d294ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d54581-266r9" Nov 02 23:44:51.307258 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:51.307240 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a54093d-f5a1-41af-b5db-deb571d294ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2d54581-266r9\" (UID: \"1a54093d-f5a1-41af-b5db-deb571d294ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d54581-266r9" Nov 02 23:44:51.309052 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:51.309022 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a54093d-f5a1-41af-b5db-deb571d294ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2d54581-266r9\" (UID: \"1a54093d-f5a1-41af-b5db-deb571d294ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d54581-266r9" Nov 02 23:44:51.367010 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:51.366975 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g78b4\" (UniqueName: \"kubernetes.io/projected/1a54093d-f5a1-41af-b5db-deb571d294ed-kube-api-access-g78b4\") pod \"maas-api-revoke-sub-a2d54581-266r9\" (UID: \"1a54093d-f5a1-41af-b5db-deb571d294ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d54581-266r9" Nov 02 23:44:51.436300 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:51.436199 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d54581-266r9" Nov 02 23:44:51.814252 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:51.814223 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2d54581-266r9"] Nov 02 23:44:51.816336 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:44:51.816306 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a54093d_f5a1_41af_b5db_deb571d294ed.slice/crio-93f8bea19b30596c7ebdc5f583404b16463dee5e008f75a52a1b33742cae14e1 WatchSource:0}: Error finding container 93f8bea19b30596c7ebdc5f583404b16463dee5e008f75a52a1b33742cae14e1: Status 404 returned error can't find the container with id 93f8bea19b30596c7ebdc5f583404b16463dee5e008f75a52a1b33742cae14e1 Nov 02 23:44:52.447073 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:52.446979 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8"] Nov 02 23:44:52.681308 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:52.681269 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d54581-266r9" event={"ID":"1a54093d-f5a1-41af-b5db-deb571d294ed","Type":"ContainerStarted","Data":"03fa83d65492bc6a39bb75e81e07a93cadb574c4bf08bde6db3d7418f41edc05"} Nov 02 23:44:52.681528 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:52.681336 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d54581-266r9" event={"ID":"1a54093d-f5a1-41af-b5db-deb571d294ed","Type":"ContainerStarted","Data":"93f8bea19b30596c7ebdc5f583404b16463dee5e008f75a52a1b33742cae14e1"} Nov 02 23:44:52.798072 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:52.798013 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d54581-266r9" podStartSLOduration=1.797996737 podStartE2EDuration="1.797996737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:44:52.7968746 +0000 UTC m=+3368.149907976" watchObservedRunningTime="2026-11-02 23:44:52.797996737 +0000 UTC m=+3368.151030112" Nov 02 23:44:53.690190 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:53.690157 5114 generic.go:356] "Generic (PLEG): container finished" podID="1a54093d-f5a1-41af-b5db-deb571d294ed" containerID="03fa83d65492bc6a39bb75e81e07a93cadb574c4bf08bde6db3d7418f41edc05" exitCode=0 Nov 02 23:44:53.690766 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:53.690249 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d54581-266r9" event={"ID":"1a54093d-f5a1-41af-b5db-deb571d294ed","Type":"ContainerDied","Data":"03fa83d65492bc6a39bb75e81e07a93cadb574c4bf08bde6db3d7418f41edc05"} Nov 02 23:44:54.841666 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:54.841639 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d54581-266r9" Nov 02 23:44:54.946107 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:54.946009 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a54093d-f5a1-41af-b5db-deb571d294ed-kube-api-access-g78b4\" (UniqueName: \"kubernetes.io/projected/1a54093d-f5a1-41af-b5db-deb571d294ed-kube-api-access-g78b4\") pod \"1a54093d-f5a1-41af-b5db-deb571d294ed\" (UID: \"1a54093d-f5a1-41af-b5db-deb571d294ed\") " Nov 02 23:44:54.946107 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:54.946105 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a54093d-f5a1-41af-b5db-deb571d294ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a54093d-f5a1-41af-b5db-deb571d294ed-maas-api-openshift-ca\") pod \"1a54093d-f5a1-41af-b5db-deb571d294ed\" (UID: \"1a54093d-f5a1-41af-b5db-deb571d294ed\") " Nov 02 23:44:54.946329 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:54.946152 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a54093d-f5a1-41af-b5db-deb571d294ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a54093d-f5a1-41af-b5db-deb571d294ed-maas-api-xks-ca\") pod \"1a54093d-f5a1-41af-b5db-deb571d294ed\" (UID: \"1a54093d-f5a1-41af-b5db-deb571d294ed\") " Nov 02 23:44:54.946537 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:54.946511 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a54093d-f5a1-41af-b5db-deb571d294ed-maas-api-openshift-ca" pod "1a54093d-f5a1-41af-b5db-deb571d294ed" (UID: "1a54093d-f5a1-41af-b5db-deb571d294ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:44:54.948524 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:54.948494 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a54093d-f5a1-41af-b5db-deb571d294ed-maas-api-xks-ca" pod "1a54093d-f5a1-41af-b5db-deb571d294ed" (UID: "1a54093d-f5a1-41af-b5db-deb571d294ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:44:54.948645 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:54.948630 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a54093d-f5a1-41af-b5db-deb571d294ed-kube-api-access-g78b4" pod "1a54093d-f5a1-41af-b5db-deb571d294ed" (UID: "1a54093d-f5a1-41af-b5db-deb571d294ed"). InnerVolumeSpecName "kube-api-access-g78b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:44:55.047446 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:55.047407 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g78b4\" (UniqueName: \"kubernetes.io/projected/1a54093d-f5a1-41af-b5db-deb571d294ed-kube-api-access-g78b4\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:44:55.047446 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:55.047440 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a54093d-f5a1-41af-b5db-deb571d294ed-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:44:55.047446 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:55.047452 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a54093d-f5a1-41af-b5db-deb571d294ed-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:44:55.703105 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:55.703076 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d54581-266r9" Nov 02 23:44:55.703288 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:55.703115 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d54581-266r9" event={"ID":"1a54093d-f5a1-41af-b5db-deb571d294ed","Type":"ContainerDied","Data":"93f8bea19b30596c7ebdc5f583404b16463dee5e008f75a52a1b33742cae14e1"} Nov 02 23:44:55.703288 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:44:55.703166 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93f8bea19b30596c7ebdc5f583404b16463dee5e008f75a52a1b33742cae14e1" Nov 02 23:45:00.156763 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.156724 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w"] Nov 02 23:45:00.157290 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.157272 5114 cpu_manager.go:414] "containerMap: removing container" podUID="1a54093d-f5a1-41af-b5db-deb571d294ed" containerName="revoke-keys" Nov 02 23:45:00.157335 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.157295 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a54093d-f5a1-41af-b5db-deb571d294ed" containerName="revoke-keys" Nov 02 23:45:00.157403 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.157393 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a54093d-f5a1-41af-b5db-deb571d294ed" containerName="revoke-keys" Nov 02 23:45:00.160953 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.160900 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w" Nov 02 23:45:00.163632 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.163607 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48"] Nov 02 23:45:00.167188 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.167167 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq"] Nov 02 23:45:00.167332 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.167311 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48" Nov 02 23:45:00.170988 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.170967 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq" Nov 02 23:45:00.203067 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.203028 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2ptt\" (UniqueName: \"kubernetes.io/projected/384a2134-0a3a-4f97-9848-02193a94df3a-kube-api-access-z2ptt\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w\" (UID: \"384a2134-0a3a-4f97-9848-02193a94df3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w" Nov 02 23:45:00.203271 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.203218 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18f144a-0ec3-4920-b630-a34e6959fac8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48\" (UID: \"b18f144a-0ec3-4920-b630-a34e6959fac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48" Nov 02 23:45:00.203367 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.203294 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnjlw\" (UniqueName: \"kubernetes.io/projected/b18f144a-0ec3-4920-b630-a34e6959fac8-kube-api-access-fnjlw\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48\" (UID: \"b18f144a-0ec3-4920-b630-a34e6959fac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48" Nov 02 23:45:00.203367 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.203339 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/384a2134-0a3a-4f97-9848-02193a94df3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w\" (UID: \"384a2134-0a3a-4f97-9848-02193a94df3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w" Nov 02 23:45:00.203520 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.203461 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b77nm\" (UniqueName: \"kubernetes.io/projected/33912f7f-d988-447b-b49d-b2af9b285b39-kube-api-access-b77nm\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq\" (UID: \"33912f7f-d988-447b-b49d-b2af9b285b39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq" Nov 02 23:45:00.203594 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.203536 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33912f7f-d988-447b-b49d-b2af9b285b39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq\" (UID: \"33912f7f-d988-447b-b49d-b2af9b285b39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq" Nov 02 23:45:00.206878 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.206854 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv"] Nov 02 23:45:00.211048 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.211023 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns"] Nov 02 23:45:00.211189 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.211134 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv" Nov 02 23:45:00.212878 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.212856 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:45:00.214380 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.214362 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr"] Nov 02 23:45:00.214503 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.214494 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns" Nov 02 23:45:00.217773 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.217756 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl"] Nov 02 23:45:00.217869 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.217860 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr" Nov 02 23:45:00.221146 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.221130 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl" Nov 02 23:45:00.266175 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.266141 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w"] Nov 02 23:45:00.268277 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.268256 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48"] Nov 02 23:45:00.305321 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.305278 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r8jd\" (UniqueName: \"kubernetes.io/projected/55787ff7-00ec-4047-a21b-71cc26ccd553-kube-api-access-5r8jd\") pod \"maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr\" (UID: \"55787ff7-00ec-4047-a21b-71cc26ccd553\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr" Nov 02 23:45:00.305575 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.305344 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg75q\" (UniqueName: \"kubernetes.io/projected/7598e2a8-ab19-48ca-b107-94451972d5bf-kube-api-access-rg75q\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv\" (UID: \"7598e2a8-ab19-48ca-b107-94451972d5bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv" Nov 02 23:45:00.305575 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.305402 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18f144a-0ec3-4920-b630-a34e6959fac8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48\" (UID: \"b18f144a-0ec3-4920-b630-a34e6959fac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48" Nov 02 23:45:00.305575 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.305471 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnjlw\" (UniqueName: \"kubernetes.io/projected/b18f144a-0ec3-4920-b630-a34e6959fac8-kube-api-access-fnjlw\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48\" (UID: \"b18f144a-0ec3-4920-b630-a34e6959fac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48" Nov 02 23:45:00.305575 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.305557 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/384a2134-0a3a-4f97-9848-02193a94df3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w\" (UID: \"384a2134-0a3a-4f97-9848-02193a94df3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w" Nov 02 23:45:00.305878 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.305590 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55787ff7-00ec-4047-a21b-71cc26ccd553-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr\" (UID: \"55787ff7-00ec-4047-a21b-71cc26ccd553\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr" Nov 02 23:45:00.305878 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.305630 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b77nm\" (UniqueName: \"kubernetes.io/projected/33912f7f-d988-447b-b49d-b2af9b285b39-kube-api-access-b77nm\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq\" (UID: \"33912f7f-d988-447b-b49d-b2af9b285b39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq" Nov 02 23:45:00.305878 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.305677 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33912f7f-d988-447b-b49d-b2af9b285b39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq\" (UID: \"33912f7f-d988-447b-b49d-b2af9b285b39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq" Nov 02 23:45:00.305878 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.305717 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7598e2a8-ab19-48ca-b107-94451972d5bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv\" (UID: \"7598e2a8-ab19-48ca-b107-94451972d5bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv" Nov 02 23:45:00.305878 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.305773 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/476aca9f-b87e-4ce3-a6d6-ea20e6bb7686-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl\" (UID: \"476aca9f-b87e-4ce3-a6d6-ea20e6bb7686\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl" Nov 02 23:45:00.306123 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.305808 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8hlc\" (UniqueName: \"kubernetes.io/projected/476aca9f-b87e-4ce3-a6d6-ea20e6bb7686-kube-api-access-m8hlc\") pod \"maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl\" (UID: \"476aca9f-b87e-4ce3-a6d6-ea20e6bb7686\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl" Nov 02 23:45:00.306174 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.306151 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2ptt\" (UniqueName: \"kubernetes.io/projected/384a2134-0a3a-4f97-9848-02193a94df3a-kube-api-access-z2ptt\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w\" (UID: \"384a2134-0a3a-4f97-9848-02193a94df3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w" Nov 02 23:45:00.306254 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.306231 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp8fv\" (UniqueName: \"kubernetes.io/projected/20959fb5-d0a9-405f-bdf6-0b778ef9fc33-kube-api-access-hp8fv\") pod \"maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns\" (UID: \"20959fb5-d0a9-405f-bdf6-0b778ef9fc33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns" Nov 02 23:45:00.306521 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.306464 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20959fb5-d0a9-405f-bdf6-0b778ef9fc33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns\" (UID: \"20959fb5-d0a9-405f-bdf6-0b778ef9fc33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns" Nov 02 23:45:00.310107 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.310075 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq"] Nov 02 23:45:00.313904 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.311211 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33912f7f-d988-447b-b49d-b2af9b285b39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq\" (UID: \"33912f7f-d988-447b-b49d-b2af9b285b39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq" Nov 02 23:45:00.313904 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.311884 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18f144a-0ec3-4920-b630-a34e6959fac8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48\" (UID: \"b18f144a-0ec3-4920-b630-a34e6959fac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48" Nov 02 23:45:00.322506 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.320168 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv"] Nov 02 23:45:00.322506 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.321513 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/384a2134-0a3a-4f97-9848-02193a94df3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w\" (UID: \"384a2134-0a3a-4f97-9848-02193a94df3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w" Nov 02 23:45:00.326507 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.322742 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns"] Nov 02 23:45:00.326507 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.325800 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr"] Nov 02 23:45:00.368741 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.368669 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl"] Nov 02 23:45:00.372601 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.372566 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b77nm\" (UniqueName: \"kubernetes.io/projected/33912f7f-d988-447b-b49d-b2af9b285b39-kube-api-access-b77nm\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq\" (UID: \"33912f7f-d988-447b-b49d-b2af9b285b39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq" Nov 02 23:45:00.373191 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.373165 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2ptt\" (UniqueName: \"kubernetes.io/projected/384a2134-0a3a-4f97-9848-02193a94df3a-kube-api-access-z2ptt\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w\" (UID: \"384a2134-0a3a-4f97-9848-02193a94df3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w" Nov 02 23:45:00.373356 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.373335 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnjlw\" (UniqueName: \"kubernetes.io/projected/b18f144a-0ec3-4920-b630-a34e6959fac8-kube-api-access-fnjlw\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48\" (UID: \"b18f144a-0ec3-4920-b630-a34e6959fac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48" Nov 02 23:45:00.408279 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.408178 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/476aca9f-b87e-4ce3-a6d6-ea20e6bb7686-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl\" (UID: \"476aca9f-b87e-4ce3-a6d6-ea20e6bb7686\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl" Nov 02 23:45:00.408279 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.408219 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8hlc\" (UniqueName: \"kubernetes.io/projected/476aca9f-b87e-4ce3-a6d6-ea20e6bb7686-kube-api-access-m8hlc\") pod \"maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl\" (UID: \"476aca9f-b87e-4ce3-a6d6-ea20e6bb7686\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl" Nov 02 23:45:00.408279 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.408267 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp8fv\" (UniqueName: \"kubernetes.io/projected/20959fb5-d0a9-405f-bdf6-0b778ef9fc33-kube-api-access-hp8fv\") pod \"maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns\" (UID: \"20959fb5-d0a9-405f-bdf6-0b778ef9fc33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns" Nov 02 23:45:00.408579 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.408287 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20959fb5-d0a9-405f-bdf6-0b778ef9fc33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns\" (UID: \"20959fb5-d0a9-405f-bdf6-0b778ef9fc33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns" Nov 02 23:45:00.408579 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.408321 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r8jd\" (UniqueName: \"kubernetes.io/projected/55787ff7-00ec-4047-a21b-71cc26ccd553-kube-api-access-5r8jd\") pod \"maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr\" (UID: \"55787ff7-00ec-4047-a21b-71cc26ccd553\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr" Nov 02 23:45:00.408579 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.408356 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg75q\" (UniqueName: \"kubernetes.io/projected/7598e2a8-ab19-48ca-b107-94451972d5bf-kube-api-access-rg75q\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv\" (UID: \"7598e2a8-ab19-48ca-b107-94451972d5bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv" Nov 02 23:45:00.409100 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.408469 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55787ff7-00ec-4047-a21b-71cc26ccd553-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr\" (UID: \"55787ff7-00ec-4047-a21b-71cc26ccd553\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr" Nov 02 23:45:00.409177 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.409153 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20959fb5-d0a9-405f-bdf6-0b778ef9fc33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns\" (UID: \"20959fb5-d0a9-405f-bdf6-0b778ef9fc33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns" Nov 02 23:45:00.409241 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.409050 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/476aca9f-b87e-4ce3-a6d6-ea20e6bb7686-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl\" (UID: \"476aca9f-b87e-4ce3-a6d6-ea20e6bb7686\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl" Nov 02 23:45:00.409241 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.409161 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7598e2a8-ab19-48ca-b107-94451972d5bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv\" (UID: \"7598e2a8-ab19-48ca-b107-94451972d5bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv" Nov 02 23:45:00.409241 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.409212 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55787ff7-00ec-4047-a21b-71cc26ccd553-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr\" (UID: \"55787ff7-00ec-4047-a21b-71cc26ccd553\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr" Nov 02 23:45:00.409653 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.409635 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7598e2a8-ab19-48ca-b107-94451972d5bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv\" (UID: \"7598e2a8-ab19-48ca-b107-94451972d5bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv" Nov 02 23:45:00.469257 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.469226 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8hlc\" (UniqueName: \"kubernetes.io/projected/476aca9f-b87e-4ce3-a6d6-ea20e6bb7686-kube-api-access-m8hlc\") pod \"maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl\" (UID: \"476aca9f-b87e-4ce3-a6d6-ea20e6bb7686\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl" Nov 02 23:45:00.469433 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.469288 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r8jd\" (UniqueName: \"kubernetes.io/projected/55787ff7-00ec-4047-a21b-71cc26ccd553-kube-api-access-5r8jd\") pod \"maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr\" (UID: \"55787ff7-00ec-4047-a21b-71cc26ccd553\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr" Nov 02 23:45:00.469433 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.469311 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp8fv\" (UniqueName: \"kubernetes.io/projected/20959fb5-d0a9-405f-bdf6-0b778ef9fc33-kube-api-access-hp8fv\") pod \"maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns\" (UID: \"20959fb5-d0a9-405f-bdf6-0b778ef9fc33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns" Nov 02 23:45:00.469433 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.469349 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg75q\" (UniqueName: \"kubernetes.io/projected/7598e2a8-ab19-48ca-b107-94451972d5bf-kube-api-access-rg75q\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv\" (UID: \"7598e2a8-ab19-48ca-b107-94451972d5bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv" Nov 02 23:45:00.477706 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.477671 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w" Nov 02 23:45:00.486336 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.486311 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48" Nov 02 23:45:00.492528 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.492504 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq" Nov 02 23:45:00.526337 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.526295 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv" Nov 02 23:45:00.535599 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.535227 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns" Nov 02 23:45:00.542307 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.542282 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr" Nov 02 23:45:00.548378 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.548345 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl" Nov 02 23:45:00.863360 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.863320 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w"] Nov 02 23:45:00.864968 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:45:00.864937 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod384a2134_0a3a_4f97_9848_02193a94df3a.slice/crio-09ab796bf2368d26ff00b3f6bf9402af7ca9c8b546df39fdf7fe8e3edbaa2ca2 WatchSource:0}: Error finding container 09ab796bf2368d26ff00b3f6bf9402af7ca9c8b546df39fdf7fe8e3edbaa2ca2: Status 404 returned error can't find the container with id 09ab796bf2368d26ff00b3f6bf9402af7ca9c8b546df39fdf7fe8e3edbaa2ca2 Nov 02 23:45:00.912962 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.912937 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48"] Nov 02 23:45:00.915417 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.915391 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq"] Nov 02 23:45:00.968363 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.968326 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns"] Nov 02 23:45:00.980282 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.980256 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv"] Nov 02 23:45:00.995165 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:00.995138 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr"] Nov 02 23:45:01.001169 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:01.001142 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl"] Nov 02 23:45:01.007947 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:45:01.007912 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18f144a_0ec3_4920_b630_a34e6959fac8.slice/crio-07591fbd5ace4906ebd3729ca9a241bd499cad16855267308c248beb7b27faa6 WatchSource:0}: Error finding container 07591fbd5ace4906ebd3729ca9a241bd499cad16855267308c248beb7b27faa6: Status 404 returned error can't find the container with id 07591fbd5ace4906ebd3729ca9a241bd499cad16855267308c248beb7b27faa6 Nov 02 23:45:01.008654 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:45:01.008631 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33912f7f_d988_447b_b49d_b2af9b285b39.slice/crio-892be23df0e54152d3be94cc94cf48faba9ee5e01b1efa6a7b4a4c9051f73efe WatchSource:0}: Error finding container 892be23df0e54152d3be94cc94cf48faba9ee5e01b1efa6a7b4a4c9051f73efe: Status 404 returned error can't find the container with id 892be23df0e54152d3be94cc94cf48faba9ee5e01b1efa6a7b4a4c9051f73efe Nov 02 23:45:01.009889 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:45:01.009765 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20959fb5_d0a9_405f_bdf6_0b778ef9fc33.slice/crio-6a8c331f471d061b1a18191c493113cf5c26ac51127f63f2412719417e859250 WatchSource:0}: Error finding container 6a8c331f471d061b1a18191c493113cf5c26ac51127f63f2412719417e859250: Status 404 returned error can't find the container with id 6a8c331f471d061b1a18191c493113cf5c26ac51127f63f2412719417e859250 Nov 02 23:45:01.010819 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:45:01.010798 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7598e2a8_ab19_48ca_b107_94451972d5bf.slice/crio-d323d567428c664d3401805f1805f42c1f5e5013afecde2990d08f4cec0659ee WatchSource:0}: Error finding container d323d567428c664d3401805f1805f42c1f5e5013afecde2990d08f4cec0659ee: Status 404 returned error can't find the container with id d323d567428c664d3401805f1805f42c1f5e5013afecde2990d08f4cec0659ee Nov 02 23:45:01.011877 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:45:01.011845 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55787ff7_00ec_4047_a21b_71cc26ccd553.slice/crio-61e4daa7828118cbc3ca02aa8ab4470327565e486397fd75b27038a5a21ddd61 WatchSource:0}: Error finding container 61e4daa7828118cbc3ca02aa8ab4470327565e486397fd75b27038a5a21ddd61: Status 404 returned error can't find the container with id 61e4daa7828118cbc3ca02aa8ab4470327565e486397fd75b27038a5a21ddd61 Nov 02 23:45:01.739270 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:01.739219 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48" event={"ID":"b18f144a-0ec3-4920-b630-a34e6959fac8","Type":"ContainerStarted","Data":"c962fed60e2b21a72026bf67dbd7ac302039e9ea15542daa5f77bed8fa4f65ed"} Nov 02 23:45:01.739779 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:01.739316 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48" event={"ID":"b18f144a-0ec3-4920-b630-a34e6959fac8","Type":"ContainerStarted","Data":"07591fbd5ace4906ebd3729ca9a241bd499cad16855267308c248beb7b27faa6"} Nov 02 23:45:01.740850 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:01.740824 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl" event={"ID":"476aca9f-b87e-4ce3-a6d6-ea20e6bb7686","Type":"ContainerStarted","Data":"7275866ec9b72661b9dab5bf6324b6c3253d6ddef50c78ce22dec47aee410abd"} Nov 02 23:45:01.740951 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:01.740874 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl" event={"ID":"476aca9f-b87e-4ce3-a6d6-ea20e6bb7686","Type":"ContainerStarted","Data":"5674cbb475c9bfaba34ce4fce9a6b9f2a25cd53be83605ad067f5f12b40bb2cc"} Nov 02 23:45:01.742403 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:01.742377 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w" event={"ID":"384a2134-0a3a-4f97-9848-02193a94df3a","Type":"ContainerStarted","Data":"900d052345b0b9e4cebccae0b6961323001a4a561b3142cb8d974798e08ffaf2"} Nov 02 23:45:01.742551 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:01.742417 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w" event={"ID":"384a2134-0a3a-4f97-9848-02193a94df3a","Type":"ContainerStarted","Data":"09ab796bf2368d26ff00b3f6bf9402af7ca9c8b546df39fdf7fe8e3edbaa2ca2"} Nov 02 23:45:01.743982 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:01.743954 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr" event={"ID":"55787ff7-00ec-4047-a21b-71cc26ccd553","Type":"ContainerStarted","Data":"dade2a8d52735af042095eb14456c9e5a8af9dcfa1b10f41127f7c1a13faaf81"} Nov 02 23:45:01.744081 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:01.743996 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr" event={"ID":"55787ff7-00ec-4047-a21b-71cc26ccd553","Type":"ContainerStarted","Data":"61e4daa7828118cbc3ca02aa8ab4470327565e486397fd75b27038a5a21ddd61"} Nov 02 23:45:01.745593 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:01.745571 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns" event={"ID":"20959fb5-d0a9-405f-bdf6-0b778ef9fc33","Type":"ContainerStarted","Data":"e0c79984f7f4a201a9134e6e9d467681450c5f58997ccfa033464efd80b81a00"} Nov 02 23:45:01.745684 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:01.745616 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns" event={"ID":"20959fb5-d0a9-405f-bdf6-0b778ef9fc33","Type":"ContainerStarted","Data":"6a8c331f471d061b1a18191c493113cf5c26ac51127f63f2412719417e859250"} Nov 02 23:45:01.747215 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:01.747185 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv" event={"ID":"7598e2a8-ab19-48ca-b107-94451972d5bf","Type":"ContainerStarted","Data":"644908478e2a1b09474accd55dec4c27420fb08d715300693a668d1d46f27208"} Nov 02 23:45:01.747321 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:01.747229 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv" event={"ID":"7598e2a8-ab19-48ca-b107-94451972d5bf","Type":"ContainerStarted","Data":"d323d567428c664d3401805f1805f42c1f5e5013afecde2990d08f4cec0659ee"} Nov 02 23:45:01.748637 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:01.748617 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq" event={"ID":"33912f7f-d988-447b-b49d-b2af9b285b39","Type":"ContainerStarted","Data":"4405260b1fae4caf49db9d4e5476500f45e17c2cca225d0c4665c8b0b889773c"} Nov 02 23:45:01.748725 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:01.748651 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq" event={"ID":"33912f7f-d988-447b-b49d-b2af9b285b39","Type":"ContainerStarted","Data":"892be23df0e54152d3be94cc94cf48faba9ee5e01b1efa6a7b4a4c9051f73efe"} Nov 02 23:45:01.995859 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:01.995802 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns" podStartSLOduration=1.9957875390000002 podStartE2EDuration="1.995787539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:01.994564231 +0000 UTC m=+3377.347597636" watchObservedRunningTime="2026-11-02 23:45:01.995787539 +0000 UTC m=+3377.348820913" Nov 02 23:45:01.996078 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:01.996057 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48" podStartSLOduration=1.996051226 podStartE2EDuration="1.996051226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:01.870042884 +0000 UTC m=+3377.223076237" watchObservedRunningTime="2026-11-02 23:45:01.996051226 +0000 UTC m=+3377.349084610" Nov 02 23:45:02.114699 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.111613 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl" podStartSLOduration=2.11159469 podStartE2EDuration="2.11159469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:02.109883214 +0000 UTC m=+3377.462916603" watchObservedRunningTime="2026-11-02 23:45:02.11159469 +0000 UTC m=+3377.464628064" Nov 02 23:45:02.224562 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.224509 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w" podStartSLOduration=2.224463501 podStartE2EDuration="2.224463501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:02.223668744 +0000 UTC m=+3377.576702120" watchObservedRunningTime="2026-11-02 23:45:02.224463501 +0000 UTC m=+3377.577496876" Nov 02 23:45:02.339399 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.339338 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq" podStartSLOduration=2.339321205 podStartE2EDuration="2.339321205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:02.337820584 +0000 UTC m=+3377.690853972" watchObservedRunningTime="2026-11-02 23:45:02.339321205 +0000 UTC m=+3377.692354627" Nov 02 23:45:02.452781 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.452706 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv" podStartSLOduration=2.452686233 podStartE2EDuration="2.452686233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:02.452092957 +0000 UTC m=+3377.805126333" watchObservedRunningTime="2026-11-02 23:45:02.452686233 +0000 UTC m=+3377.805719608" Nov 02 23:45:02.567351 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.567284 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr" podStartSLOduration=2.567269544 podStartE2EDuration="2.567269544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:02.565697247 +0000 UTC m=+3377.918730623" watchObservedRunningTime="2026-11-02 23:45:02.567269544 +0000 UTC m=+3377.920302916" Nov 02 23:45:02.756963 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.756921 5114 generic.go:356] "Generic (PLEG): container finished" podID="33912f7f-d988-447b-b49d-b2af9b285b39" containerID="4405260b1fae4caf49db9d4e5476500f45e17c2cca225d0c4665c8b0b889773c" exitCode=60 Nov 02 23:45:02.757504 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.756977 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq" event={"ID":"33912f7f-d988-447b-b49d-b2af9b285b39","Type":"ContainerDied","Data":"4405260b1fae4caf49db9d4e5476500f45e17c2cca225d0c4665c8b0b889773c"} Nov 02 23:45:02.757504 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.757362 5114 scope.go:122] "RemoveContainer" containerID="4405260b1fae4caf49db9d4e5476500f45e17c2cca225d0c4665c8b0b889773c" Nov 02 23:45:02.758831 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.758804 5114 generic.go:356] "Generic (PLEG): container finished" podID="b18f144a-0ec3-4920-b630-a34e6959fac8" containerID="c962fed60e2b21a72026bf67dbd7ac302039e9ea15542daa5f77bed8fa4f65ed" exitCode=60 Nov 02 23:45:02.758941 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.758879 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48" event={"ID":"b18f144a-0ec3-4920-b630-a34e6959fac8","Type":"ContainerDied","Data":"c962fed60e2b21a72026bf67dbd7ac302039e9ea15542daa5f77bed8fa4f65ed"} Nov 02 23:45:02.759157 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.759133 5114 scope.go:122] "RemoveContainer" containerID="c962fed60e2b21a72026bf67dbd7ac302039e9ea15542daa5f77bed8fa4f65ed" Nov 02 23:45:02.760685 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.760661 5114 generic.go:356] "Generic (PLEG): container finished" podID="476aca9f-b87e-4ce3-a6d6-ea20e6bb7686" containerID="7275866ec9b72661b9dab5bf6324b6c3253d6ddef50c78ce22dec47aee410abd" exitCode=60 Nov 02 23:45:02.760860 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.760754 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl" event={"ID":"476aca9f-b87e-4ce3-a6d6-ea20e6bb7686","Type":"ContainerDied","Data":"7275866ec9b72661b9dab5bf6324b6c3253d6ddef50c78ce22dec47aee410abd"} Nov 02 23:45:02.761062 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.761044 5114 scope.go:122] "RemoveContainer" containerID="7275866ec9b72661b9dab5bf6324b6c3253d6ddef50c78ce22dec47aee410abd" Nov 02 23:45:02.762867 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.762844 5114 generic.go:356] "Generic (PLEG): container finished" podID="384a2134-0a3a-4f97-9848-02193a94df3a" containerID="900d052345b0b9e4cebccae0b6961323001a4a561b3142cb8d974798e08ffaf2" exitCode=60 Nov 02 23:45:02.762946 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.762924 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w" event={"ID":"384a2134-0a3a-4f97-9848-02193a94df3a","Type":"ContainerDied","Data":"900d052345b0b9e4cebccae0b6961323001a4a561b3142cb8d974798e08ffaf2"} Nov 02 23:45:02.763116 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.763101 5114 scope.go:122] "RemoveContainer" containerID="900d052345b0b9e4cebccae0b6961323001a4a561b3142cb8d974798e08ffaf2" Nov 02 23:45:02.764604 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.764579 5114 generic.go:356] "Generic (PLEG): container finished" podID="55787ff7-00ec-4047-a21b-71cc26ccd553" containerID="dade2a8d52735af042095eb14456c9e5a8af9dcfa1b10f41127f7c1a13faaf81" exitCode=60 Nov 02 23:45:02.764690 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.764665 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr" event={"ID":"55787ff7-00ec-4047-a21b-71cc26ccd553","Type":"ContainerDied","Data":"dade2a8d52735af042095eb14456c9e5a8af9dcfa1b10f41127f7c1a13faaf81"} Nov 02 23:45:02.764848 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.764831 5114 scope.go:122] "RemoveContainer" containerID="dade2a8d52735af042095eb14456c9e5a8af9dcfa1b10f41127f7c1a13faaf81" Nov 02 23:45:02.766380 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.766362 5114 generic.go:356] "Generic (PLEG): container finished" podID="20959fb5-d0a9-405f-bdf6-0b778ef9fc33" containerID="e0c79984f7f4a201a9134e6e9d467681450c5f58997ccfa033464efd80b81a00" exitCode=60 Nov 02 23:45:02.766472 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.766449 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns" event={"ID":"20959fb5-d0a9-405f-bdf6-0b778ef9fc33","Type":"ContainerDied","Data":"e0c79984f7f4a201a9134e6e9d467681450c5f58997ccfa033464efd80b81a00"} Nov 02 23:45:02.766744 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.766727 5114 scope.go:122] "RemoveContainer" containerID="e0c79984f7f4a201a9134e6e9d467681450c5f58997ccfa033464efd80b81a00" Nov 02 23:45:02.768757 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.768739 5114 generic.go:356] "Generic (PLEG): container finished" podID="7598e2a8-ab19-48ca-b107-94451972d5bf" containerID="644908478e2a1b09474accd55dec4c27420fb08d715300693a668d1d46f27208" exitCode=60 Nov 02 23:45:02.769253 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.768780 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv" event={"ID":"7598e2a8-ab19-48ca-b107-94451972d5bf","Type":"ContainerDied","Data":"644908478e2a1b09474accd55dec4c27420fb08d715300693a668d1d46f27208"} Nov 02 23:45:02.769290 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:02.769062 5114 scope.go:122] "RemoveContainer" containerID="644908478e2a1b09474accd55dec4c27420fb08d715300693a668d1d46f27208" Nov 02 23:45:03.778450 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:03.778350 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq" event={"ID":"33912f7f-d988-447b-b49d-b2af9b285b39","Type":"ContainerStarted","Data":"45f065ce479a450abd66788ded1fc8cffdc48c430c4c104e3c6ce63082927195"} Nov 02 23:45:03.780517 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:03.780460 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48" event={"ID":"b18f144a-0ec3-4920-b630-a34e6959fac8","Type":"ContainerStarted","Data":"5d0f2c2ef1a9282ba4841cf8995921219a73ba4d6096c77e563ce4858b51cc0c"} Nov 02 23:45:03.782439 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:03.782404 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl" event={"ID":"476aca9f-b87e-4ce3-a6d6-ea20e6bb7686","Type":"ContainerStarted","Data":"91992fb3bc72d162ae94a074e1acabafc7937a47f6a0281d2a90a0b135374b89"} Nov 02 23:45:03.785637 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:03.785605 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w" event={"ID":"384a2134-0a3a-4f97-9848-02193a94df3a","Type":"ContainerStarted","Data":"b62dc79b07699c1491e800d22a2715b9c31a1790e665d6c6770555ae961cfaee"} Nov 02 23:45:03.788392 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:03.788032 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr" event={"ID":"55787ff7-00ec-4047-a21b-71cc26ccd553","Type":"ContainerStarted","Data":"b0a0fee572f3047ff565df86e13aa0128bc2b750b90a8d3b343b5607a7aa0e68"} Nov 02 23:45:03.790213 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:03.790185 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns" event={"ID":"20959fb5-d0a9-405f-bdf6-0b778ef9fc33","Type":"ContainerStarted","Data":"8fafc7d3820d7261f20cf2aa2e47a63516ca9c1e7a20717e4ce9b9faa4194ac3"} Nov 02 23:45:03.792162 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:03.792138 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv" event={"ID":"7598e2a8-ab19-48ca-b107-94451972d5bf","Type":"ContainerStarted","Data":"d5b204337cc922820fefa20fa5cc7679a70043ba469bbd4eb345fa694975e07b"} Nov 02 23:45:04.800867 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.800830 5114 generic.go:356] "Generic (PLEG): container finished" podID="b18f144a-0ec3-4920-b630-a34e6959fac8" containerID="5d0f2c2ef1a9282ba4841cf8995921219a73ba4d6096c77e563ce4858b51cc0c" exitCode=60 Nov 02 23:45:04.801367 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.800925 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48" event={"ID":"b18f144a-0ec3-4920-b630-a34e6959fac8","Type":"ContainerDied","Data":"5d0f2c2ef1a9282ba4841cf8995921219a73ba4d6096c77e563ce4858b51cc0c"} Nov 02 23:45:04.801367 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.800998 5114 scope.go:122] "RemoveContainer" containerID="c962fed60e2b21a72026bf67dbd7ac302039e9ea15542daa5f77bed8fa4f65ed" Nov 02 23:45:04.801367 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.801230 5114 scope.go:122] "RemoveContainer" containerID="5d0f2c2ef1a9282ba4841cf8995921219a73ba4d6096c77e563ce4858b51cc0c" Nov 02 23:45:04.801618 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:04.801592 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48_odh-ai-gateway-infra(b18f144a-0ec3-4920-b630-a34e6959fac8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48" podUID="b18f144a-0ec3-4920-b630-a34e6959fac8" Nov 02 23:45:04.802850 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.802826 5114 generic.go:356] "Generic (PLEG): container finished" podID="476aca9f-b87e-4ce3-a6d6-ea20e6bb7686" containerID="91992fb3bc72d162ae94a074e1acabafc7937a47f6a0281d2a90a0b135374b89" exitCode=60 Nov 02 23:45:04.802938 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.802912 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl" event={"ID":"476aca9f-b87e-4ce3-a6d6-ea20e6bb7686","Type":"ContainerDied","Data":"91992fb3bc72d162ae94a074e1acabafc7937a47f6a0281d2a90a0b135374b89"} Nov 02 23:45:04.803183 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.803163 5114 scope.go:122] "RemoveContainer" containerID="91992fb3bc72d162ae94a074e1acabafc7937a47f6a0281d2a90a0b135374b89" Nov 02 23:45:04.803526 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:04.803470 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl_odh-ai-gateway-infra(476aca9f-b87e-4ce3-a6d6-ea20e6bb7686)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl" podUID="476aca9f-b87e-4ce3-a6d6-ea20e6bb7686" Nov 02 23:45:04.804933 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.804907 5114 generic.go:356] "Generic (PLEG): container finished" podID="384a2134-0a3a-4f97-9848-02193a94df3a" containerID="b62dc79b07699c1491e800d22a2715b9c31a1790e665d6c6770555ae961cfaee" exitCode=60 Nov 02 23:45:04.805049 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.804981 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w" event={"ID":"384a2134-0a3a-4f97-9848-02193a94df3a","Type":"ContainerDied","Data":"b62dc79b07699c1491e800d22a2715b9c31a1790e665d6c6770555ae961cfaee"} Nov 02 23:45:04.805311 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.805292 5114 scope.go:122] "RemoveContainer" containerID="b62dc79b07699c1491e800d22a2715b9c31a1790e665d6c6770555ae961cfaee" Nov 02 23:45:04.805639 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:04.805618 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w_odh-ai-gateway-infra(384a2134-0a3a-4f97-9848-02193a94df3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w" podUID="384a2134-0a3a-4f97-9848-02193a94df3a" Nov 02 23:45:04.807176 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.807154 5114 generic.go:356] "Generic (PLEG): container finished" podID="55787ff7-00ec-4047-a21b-71cc26ccd553" containerID="b0a0fee572f3047ff565df86e13aa0128bc2b750b90a8d3b343b5607a7aa0e68" exitCode=60 Nov 02 23:45:04.807283 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.807238 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr" event={"ID":"55787ff7-00ec-4047-a21b-71cc26ccd553","Type":"ContainerDied","Data":"b0a0fee572f3047ff565df86e13aa0128bc2b750b90a8d3b343b5607a7aa0e68"} Nov 02 23:45:04.807527 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.807512 5114 scope.go:122] "RemoveContainer" containerID="b0a0fee572f3047ff565df86e13aa0128bc2b750b90a8d3b343b5607a7aa0e68" Nov 02 23:45:04.807821 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:04.807801 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr_odh-ai-gateway-infra(55787ff7-00ec-4047-a21b-71cc26ccd553)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr" podUID="55787ff7-00ec-4047-a21b-71cc26ccd553" Nov 02 23:45:04.809231 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.809209 5114 generic.go:356] "Generic (PLEG): container finished" podID="20959fb5-d0a9-405f-bdf6-0b778ef9fc33" containerID="8fafc7d3820d7261f20cf2aa2e47a63516ca9c1e7a20717e4ce9b9faa4194ac3" exitCode=60 Nov 02 23:45:04.809339 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.809305 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns" event={"ID":"20959fb5-d0a9-405f-bdf6-0b778ef9fc33","Type":"ContainerDied","Data":"8fafc7d3820d7261f20cf2aa2e47a63516ca9c1e7a20717e4ce9b9faa4194ac3"} Nov 02 23:45:04.809556 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.809536 5114 scope.go:122] "RemoveContainer" containerID="8fafc7d3820d7261f20cf2aa2e47a63516ca9c1e7a20717e4ce9b9faa4194ac3" Nov 02 23:45:04.809844 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:04.809819 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns_odh-ai-gateway-infra(20959fb5-d0a9-405f-bdf6-0b778ef9fc33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns" podUID="20959fb5-d0a9-405f-bdf6-0b778ef9fc33" Nov 02 23:45:04.815847 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.815814 5114 generic.go:356] "Generic (PLEG): container finished" podID="7598e2a8-ab19-48ca-b107-94451972d5bf" containerID="d5b204337cc922820fefa20fa5cc7679a70043ba469bbd4eb345fa694975e07b" exitCode=60 Nov 02 23:45:04.816005 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.815901 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv" event={"ID":"7598e2a8-ab19-48ca-b107-94451972d5bf","Type":"ContainerDied","Data":"d5b204337cc922820fefa20fa5cc7679a70043ba469bbd4eb345fa694975e07b"} Nov 02 23:45:04.816164 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.816148 5114 scope.go:122] "RemoveContainer" containerID="d5b204337cc922820fefa20fa5cc7679a70043ba469bbd4eb345fa694975e07b" Nov 02 23:45:04.816446 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:04.816428 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv_odh-ai-gateway-infra(7598e2a8-ab19-48ca-b107-94451972d5bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv" podUID="7598e2a8-ab19-48ca-b107-94451972d5bf" Nov 02 23:45:04.817027 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.817011 5114 scope.go:122] "RemoveContainer" containerID="7275866ec9b72661b9dab5bf6324b6c3253d6ddef50c78ce22dec47aee410abd" Nov 02 23:45:04.817918 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.817896 5114 generic.go:356] "Generic (PLEG): container finished" podID="33912f7f-d988-447b-b49d-b2af9b285b39" containerID="45f065ce479a450abd66788ded1fc8cffdc48c430c4c104e3c6ce63082927195" exitCode=60 Nov 02 23:45:04.818017 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.817939 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq" event={"ID":"33912f7f-d988-447b-b49d-b2af9b285b39","Type":"ContainerDied","Data":"45f065ce479a450abd66788ded1fc8cffdc48c430c4c104e3c6ce63082927195"} Nov 02 23:45:04.818246 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.818228 5114 scope.go:122] "RemoveContainer" containerID="45f065ce479a450abd66788ded1fc8cffdc48c430c4c104e3c6ce63082927195" Nov 02 23:45:04.818613 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:04.818588 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq_odh-ai-gateway-infra(33912f7f-d988-447b-b49d-b2af9b285b39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq" podUID="33912f7f-d988-447b-b49d-b2af9b285b39" Nov 02 23:45:04.830447 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.830424 5114 scope.go:122] "RemoveContainer" containerID="900d052345b0b9e4cebccae0b6961323001a4a561b3142cb8d974798e08ffaf2" Nov 02 23:45:04.848406 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.848381 5114 scope.go:122] "RemoveContainer" containerID="dade2a8d52735af042095eb14456c9e5a8af9dcfa1b10f41127f7c1a13faaf81" Nov 02 23:45:04.863132 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.863109 5114 scope.go:122] "RemoveContainer" containerID="e0c79984f7f4a201a9134e6e9d467681450c5f58997ccfa033464efd80b81a00" Nov 02 23:45:04.876687 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.876663 5114 scope.go:122] "RemoveContainer" containerID="644908478e2a1b09474accd55dec4c27420fb08d715300693a668d1d46f27208" Nov 02 23:45:04.890939 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:04.890912 5114 scope.go:122] "RemoveContainer" containerID="4405260b1fae4caf49db9d4e5476500f45e17c2cca225d0c4665c8b0b889773c" Nov 02 23:45:05.827129 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:05.827097 5114 scope.go:122] "RemoveContainer" containerID="5d0f2c2ef1a9282ba4841cf8995921219a73ba4d6096c77e563ce4858b51cc0c" Nov 02 23:45:05.827603 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:05.827437 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48_odh-ai-gateway-infra(b18f144a-0ec3-4920-b630-a34e6959fac8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48" podUID="b18f144a-0ec3-4920-b630-a34e6959fac8" Nov 02 23:45:05.828780 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:05.828752 5114 scope.go:122] "RemoveContainer" containerID="91992fb3bc72d162ae94a074e1acabafc7937a47f6a0281d2a90a0b135374b89" Nov 02 23:45:05.829051 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:05.829022 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl_odh-ai-gateway-infra(476aca9f-b87e-4ce3-a6d6-ea20e6bb7686)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl" podUID="476aca9f-b87e-4ce3-a6d6-ea20e6bb7686" Nov 02 23:45:05.830326 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:05.830303 5114 scope.go:122] "RemoveContainer" containerID="b62dc79b07699c1491e800d22a2715b9c31a1790e665d6c6770555ae961cfaee" Nov 02 23:45:05.830661 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:05.830631 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w_odh-ai-gateway-infra(384a2134-0a3a-4f97-9848-02193a94df3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w" podUID="384a2134-0a3a-4f97-9848-02193a94df3a" Nov 02 23:45:05.831999 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:05.831981 5114 scope.go:122] "RemoveContainer" containerID="b0a0fee572f3047ff565df86e13aa0128bc2b750b90a8d3b343b5607a7aa0e68" Nov 02 23:45:05.832207 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:05.832185 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr_odh-ai-gateway-infra(55787ff7-00ec-4047-a21b-71cc26ccd553)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr" podUID="55787ff7-00ec-4047-a21b-71cc26ccd553" Nov 02 23:45:05.833510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:05.833463 5114 scope.go:122] "RemoveContainer" containerID="8fafc7d3820d7261f20cf2aa2e47a63516ca9c1e7a20717e4ce9b9faa4194ac3" Nov 02 23:45:05.833774 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:05.833755 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns_odh-ai-gateway-infra(20959fb5-d0a9-405f-bdf6-0b778ef9fc33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns" podUID="20959fb5-d0a9-405f-bdf6-0b778ef9fc33" Nov 02 23:45:05.835001 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:05.834984 5114 scope.go:122] "RemoveContainer" containerID="d5b204337cc922820fefa20fa5cc7679a70043ba469bbd4eb345fa694975e07b" Nov 02 23:45:05.835220 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:05.835203 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv_odh-ai-gateway-infra(7598e2a8-ab19-48ca-b107-94451972d5bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv" podUID="7598e2a8-ab19-48ca-b107-94451972d5bf" Nov 02 23:45:05.836377 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:05.836359 5114 scope.go:122] "RemoveContainer" containerID="45f065ce479a450abd66788ded1fc8cffdc48c430c4c104e3c6ce63082927195" Nov 02 23:45:05.836578 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:05.836561 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq_odh-ai-gateway-infra(33912f7f-d988-447b-b49d-b2af9b285b39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq" podUID="33912f7f-d988-447b-b49d-b2af9b285b39" Nov 02 23:45:07.469107 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.469066 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" podUID="27440af4-08bc-4f25-a76f-41a153b347e1" containerName="main" containerID="cri-o://650b74ddb968c1e152f50568c8d784b77f9f17371d2906ca5462b5b038d552ec" gracePeriod=45 Nov 02 23:45:07.569604 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.569555 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" podUID="27440af4-08bc-4f25-a76f-41a153b347e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": dial tcp 10.134.0.96:8000: connect: connection refused" Nov 02 23:45:07.584858 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:07.584815 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:45:07.743847 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.743818 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:45:07.796444 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.796408 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27440af4-08bc-4f25-a76f-41a153b347e1-kube-api-access-9k976\" (UniqueName: \"kubernetes.io/projected/27440af4-08bc-4f25-a76f-41a153b347e1-kube-api-access-9k976\") pod \"27440af4-08bc-4f25-a76f-41a153b347e1\" (UID: \"27440af4-08bc-4f25-a76f-41a153b347e1\") " Nov 02 23:45:07.796636 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.796568 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27440af4-08bc-4f25-a76f-41a153b347e1-tls-certs\" (UniqueName: \"kubernetes.io/secret/27440af4-08bc-4f25-a76f-41a153b347e1-tls-certs\") pod \"27440af4-08bc-4f25-a76f-41a153b347e1\" (UID: \"27440af4-08bc-4f25-a76f-41a153b347e1\") " Nov 02 23:45:07.796685 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.796666 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-model-cache\") pod \"27440af4-08bc-4f25-a76f-41a153b347e1\" (UID: \"27440af4-08bc-4f25-a76f-41a153b347e1\") " Nov 02 23:45:07.796778 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.796757 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-dshm\") pod \"27440af4-08bc-4f25-a76f-41a153b347e1\" (UID: \"27440af4-08bc-4f25-a76f-41a153b347e1\") " Nov 02 23:45:07.796923 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.796804 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-tmp-dir\") pod \"27440af4-08bc-4f25-a76f-41a153b347e1\" (UID: \"27440af4-08bc-4f25-a76f-41a153b347e1\") " Nov 02 23:45:07.796923 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.796883 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-home\" (UniqueName: \"kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-home\") pod \"27440af4-08bc-4f25-a76f-41a153b347e1\" (UID: \"27440af4-08bc-4f25-a76f-41a153b347e1\") " Nov 02 23:45:07.797072 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.796941 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-model-cache" pod "27440af4-08bc-4f25-a76f-41a153b347e1" (UID: "27440af4-08bc-4f25-a76f-41a153b347e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:07.797139 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.797098 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-tmp-dir" pod "27440af4-08bc-4f25-a76f-41a153b347e1" (UID: "27440af4-08bc-4f25-a76f-41a153b347e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:07.797233 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.797215 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-home" pod "27440af4-08bc-4f25-a76f-41a153b347e1" (UID: "27440af4-08bc-4f25-a76f-41a153b347e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:07.797352 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.797336 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:07.797392 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.797369 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:07.797442 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.797395 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:07.799053 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.799026 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27440af4-08bc-4f25-a76f-41a153b347e1-tls-certs" pod "27440af4-08bc-4f25-a76f-41a153b347e1" (UID: "27440af4-08bc-4f25-a76f-41a153b347e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:45:07.799153 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.799127 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27440af4-08bc-4f25-a76f-41a153b347e1-kube-api-access-9k976" pod "27440af4-08bc-4f25-a76f-41a153b347e1" (UID: "27440af4-08bc-4f25-a76f-41a153b347e1"). InnerVolumeSpecName "kube-api-access-9k976". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:07.799325 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.799309 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-dshm" pod "27440af4-08bc-4f25-a76f-41a153b347e1" (UID: "27440af4-08bc-4f25-a76f-41a153b347e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:07.848415 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.848375 5114 generic.go:356] "Generic (PLEG): container finished" podID="27440af4-08bc-4f25-a76f-41a153b347e1" containerID="650b74ddb968c1e152f50568c8d784b77f9f17371d2906ca5462b5b038d552ec" exitCode=0 Nov 02 23:45:07.848592 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.848430 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" Nov 02 23:45:07.848592 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.848444 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" event={"ID":"27440af4-08bc-4f25-a76f-41a153b347e1","Type":"ContainerDied","Data":"650b74ddb968c1e152f50568c8d784b77f9f17371d2906ca5462b5b038d552ec"} Nov 02 23:45:07.848592 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.848498 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8" event={"ID":"27440af4-08bc-4f25-a76f-41a153b347e1","Type":"ContainerDied","Data":"3bbe98fbed07dab396a1f90736539a83b0f06cd4c0fdbd1176e87cc200b594d4"} Nov 02 23:45:07.848592 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.848521 5114 scope.go:122] "RemoveContainer" containerID="650b74ddb968c1e152f50568c8d784b77f9f17371d2906ca5462b5b038d552ec" Nov 02 23:45:07.860499 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.860363 5114 scope.go:122] "RemoveContainer" containerID="650b74ddb968c1e152f50568c8d784b77f9f17371d2906ca5462b5b038d552ec" Nov 02 23:45:07.860810 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:07.860778 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650b74ddb968c1e152f50568c8d784b77f9f17371d2906ca5462b5b038d552ec\": container with ID starting with 650b74ddb968c1e152f50568c8d784b77f9f17371d2906ca5462b5b038d552ec not found: ID does not exist" containerID="650b74ddb968c1e152f50568c8d784b77f9f17371d2906ca5462b5b038d552ec" Nov 02 23:45:07.860897 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.860819 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650b74ddb968c1e152f50568c8d784b77f9f17371d2906ca5462b5b038d552ec"} err="failed to get container status \"650b74ddb968c1e152f50568c8d784b77f9f17371d2906ca5462b5b038d552ec\": rpc error: code = NotFound desc = could not find container \"650b74ddb968c1e152f50568c8d784b77f9f17371d2906ca5462b5b038d552ec\": container with ID starting with 650b74ddb968c1e152f50568c8d784b77f9f17371d2906ca5462b5b038d552ec not found: ID does not exist" Nov 02 23:45:07.898361 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.898322 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27440af4-08bc-4f25-a76f-41a153b347e1-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:07.898361 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.898363 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k976\" (UniqueName: \"kubernetes.io/projected/27440af4-08bc-4f25-a76f-41a153b347e1-kube-api-access-9k976\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:07.898668 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.898379 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27440af4-08bc-4f25-a76f-41a153b347e1-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:07.970974 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.970934 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8"] Nov 02 23:45:07.980606 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:07.980525 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-980b82/auth-test-model-980b82-kserve-64ff84f86f-jsdq8"] Nov 02 23:45:08.793283 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:08.793238 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27440af4-08bc-4f25-a76f-41a153b347e1" path="/var/lib/kubelet/pods/27440af4-08bc-4f25-a76f-41a153b347e1/volumes" Nov 02 23:45:09.129328 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:09.129240 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz"] Nov 02 23:45:11.988281 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:11.988232 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn"] Nov 02 23:45:11.988968 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:11.988944 5114 cpu_manager.go:414] "containerMap: removing container" podUID="27440af4-08bc-4f25-a76f-41a153b347e1" containerName="main" Nov 02 23:45:11.989070 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:11.988972 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27440af4-08bc-4f25-a76f-41a153b347e1" containerName="main" Nov 02 23:45:11.989162 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:11.989148 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="27440af4-08bc-4f25-a76f-41a153b347e1" containerName="main" Nov 02 23:45:11.993991 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:11.993966 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:45:12.050231 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:12.050192 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn"] Nov 02 23:45:12.096422 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:12.096380 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-38049c\"/\"kube-root-ca.crt\"" Nov 02 23:45:12.096658 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:12.096447 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-38049c\"/\"default-dockercfg-c29j9\"" Nov 02 23:45:12.096658 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:12.096472 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-38049c\"/\"noten-model-38049c-kserve-self-signed-certs\"" Nov 02 23:45:12.096658 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:12.096519 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-38049c\"/\"openshift-service-ca.crt\"" Nov 02 23:45:12.154574 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:12.154537 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-home\") pod \"noten-model-38049c-kserve-59b767b9cc-h5fzn\" (UID: \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\") " pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:45:12.154574 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:12.154576 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-tls-certs\") pod \"noten-model-38049c-kserve-59b767b9cc-h5fzn\" (UID: \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\") " pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:45:12.154799 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:12.154732 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh4xq\" (UniqueName: \"kubernetes.io/projected/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-kube-api-access-rh4xq\") pod \"noten-model-38049c-kserve-59b767b9cc-h5fzn\" (UID: \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\") " pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:45:12.154849 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:12.154805 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-dshm\") pod \"noten-model-38049c-kserve-59b767b9cc-h5fzn\" (UID: \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\") " pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:45:12.154916 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:12.154894 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-model-cache\") pod \"noten-model-38049c-kserve-59b767b9cc-h5fzn\" (UID: \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\") " pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:45:12.154967 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:12.154936 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-tmp-dir\") pod \"noten-model-38049c-kserve-59b767b9cc-h5fzn\" (UID: \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\") " pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:45:12.255679 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:12.255574 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-dshm\") pod \"noten-model-38049c-kserve-59b767b9cc-h5fzn\" (UID: \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\") " pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:45:12.255679 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:12.255636 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-model-cache\") pod \"noten-model-38049c-kserve-59b767b9cc-h5fzn\" (UID: \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\") " pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:45:12.255679 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:12.255675 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-tmp-dir\") pod \"noten-model-38049c-kserve-59b767b9cc-h5fzn\" (UID: \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\") " pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:45:12.255991 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:12.255735 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-home\") pod \"noten-model-38049c-kserve-59b767b9cc-h5fzn\" (UID: \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\") " pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:45:12.255991 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:12.255758 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-tls-certs\") pod \"noten-model-38049c-kserve-59b767b9cc-h5fzn\" (UID: \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\") " pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:45:12.255991 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:12.255853 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh4xq\" (UniqueName: \"kubernetes.io/projected/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-kube-api-access-rh4xq\") pod \"noten-model-38049c-kserve-59b767b9cc-h5fzn\" (UID: \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\") " pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:45:12.256156 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:12.256126 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-model-cache\") pod \"noten-model-38049c-kserve-59b767b9cc-h5fzn\" (UID: \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\") " pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:45:12.256673 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:12.256647 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-home\") pod \"noten-model-38049c-kserve-59b767b9cc-h5fzn\" (UID: \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\") " pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:45:12.256838 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:12.256810 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-tmp-dir\") pod \"noten-model-38049c-kserve-59b767b9cc-h5fzn\" (UID: \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\") " pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:45:12.258528 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:12.258497 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-dshm\") pod \"noten-model-38049c-kserve-59b767b9cc-h5fzn\" (UID: \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\") " pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:45:12.259018 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:12.258994 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-tls-certs\") pod \"noten-model-38049c-kserve-59b767b9cc-h5fzn\" (UID: \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\") " pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:45:12.318901 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:12.318862 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh4xq\" (UniqueName: \"kubernetes.io/projected/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-kube-api-access-rh4xq\") pod \"noten-model-38049c-kserve-59b767b9cc-h5fzn\" (UID: \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\") " pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:45:12.606590 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:12.606546 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:45:12.988846 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:12.988812 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn"] Nov 02 23:45:12.990309 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:45:12.990286 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod195e0f31_6e9c_4861_aa00_2dcb8e5c17de.slice/crio-5d5c9ef70f2849f26146de9372e4d0a49fef615727f98ef2633c99877ef4d77b WatchSource:0}: Error finding container 5d5c9ef70f2849f26146de9372e4d0a49fef615727f98ef2633c99877ef4d77b: Status 404 returned error can't find the container with id 5d5c9ef70f2849f26146de9372e4d0a49fef615727f98ef2633c99877ef4d77b Nov 02 23:45:13.886593 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:13.886546 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" event={"ID":"195e0f31-6e9c-4861-aa00-2dcb8e5c17de","Type":"ContainerStarted","Data":"d0cfedab1e5cf11b0ad5855650a46bee2afbccd9e3adecb378e1a28c4399599f"} Nov 02 23:45:13.886593 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:13.886594 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" event={"ID":"195e0f31-6e9c-4861-aa00-2dcb8e5c17de","Type":"ContainerStarted","Data":"5d5c9ef70f2849f26146de9372e4d0a49fef615727f98ef2633c99877ef4d77b"} Nov 02 23:45:14.003313 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:14.003255 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" podStartSLOduration=3.003238016 podStartE2EDuration="3.003238016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:14.002088459 +0000 UTC m=+3389.355121839" watchObservedRunningTime="2026-11-02 23:45:14.003238016 +0000 UTC m=+3389.356271390" Nov 02 23:45:14.607429 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:14.607377 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:45:14.789721 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:14.789689 5114 scope.go:122] "RemoveContainer" containerID="b0a0fee572f3047ff565df86e13aa0128bc2b750b90a8d3b343b5607a7aa0e68" Nov 02 23:45:14.789892 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:14.789802 5114 scope.go:122] "RemoveContainer" containerID="8fafc7d3820d7261f20cf2aa2e47a63516ca9c1e7a20717e4ce9b9faa4194ac3" Nov 02 23:45:15.787148 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:15.787115 5114 scope.go:122] "RemoveContainer" containerID="91992fb3bc72d162ae94a074e1acabafc7937a47f6a0281d2a90a0b135374b89" Nov 02 23:45:15.787596 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:15.787337 5114 scope.go:122] "RemoveContainer" containerID="5d0f2c2ef1a9282ba4841cf8995921219a73ba4d6096c77e563ce4858b51cc0c" Nov 02 23:45:15.901137 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:15.901107 5114 generic.go:356] "Generic (PLEG): container finished" podID="20959fb5-d0a9-405f-bdf6-0b778ef9fc33" containerID="0c5c27ba3ea1aab34924c60b1e54d3e616af6711b9ac63fafa712b1ccb12b385" exitCode=60 Nov 02 23:45:15.901271 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:15.901178 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns" event={"ID":"20959fb5-d0a9-405f-bdf6-0b778ef9fc33","Type":"ContainerDied","Data":"0c5c27ba3ea1aab34924c60b1e54d3e616af6711b9ac63fafa712b1ccb12b385"} Nov 02 23:45:15.901271 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:15.901239 5114 scope.go:122] "RemoveContainer" containerID="8fafc7d3820d7261f20cf2aa2e47a63516ca9c1e7a20717e4ce9b9faa4194ac3" Nov 02 23:45:15.901663 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:15.901634 5114 scope.go:122] "RemoveContainer" containerID="0c5c27ba3ea1aab34924c60b1e54d3e616af6711b9ac63fafa712b1ccb12b385" Nov 02 23:45:15.901923 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:15.901904 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns_odh-ai-gateway-infra(20959fb5-d0a9-405f-bdf6-0b778ef9fc33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns" podUID="20959fb5-d0a9-405f-bdf6-0b778ef9fc33" Nov 02 23:45:15.906529 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:15.904265 5114 generic.go:356] "Generic (PLEG): container finished" podID="55787ff7-00ec-4047-a21b-71cc26ccd553" containerID="2c9e86907d32ad517ff12cf4e0d1e6bdf743259f929aed72328a416a227799f9" exitCode=60 Nov 02 23:45:15.906529 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:15.905880 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr" event={"ID":"55787ff7-00ec-4047-a21b-71cc26ccd553","Type":"ContainerDied","Data":"2c9e86907d32ad517ff12cf4e0d1e6bdf743259f929aed72328a416a227799f9"} Nov 02 23:45:15.906529 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:15.906317 5114 scope.go:122] "RemoveContainer" containerID="2c9e86907d32ad517ff12cf4e0d1e6bdf743259f929aed72328a416a227799f9" Nov 02 23:45:15.906768 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:15.906684 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr_odh-ai-gateway-infra(55787ff7-00ec-4047-a21b-71cc26ccd553)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr" podUID="55787ff7-00ec-4047-a21b-71cc26ccd553" Nov 02 23:45:15.916671 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:15.916647 5114 scope.go:122] "RemoveContainer" containerID="b0a0fee572f3047ff565df86e13aa0128bc2b750b90a8d3b343b5607a7aa0e68" Nov 02 23:45:16.786795 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:16.786768 5114 scope.go:122] "RemoveContainer" containerID="45f065ce479a450abd66788ded1fc8cffdc48c430c4c104e3c6ce63082927195" Nov 02 23:45:16.913067 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:16.913028 5114 generic.go:356] "Generic (PLEG): container finished" podID="b18f144a-0ec3-4920-b630-a34e6959fac8" containerID="dd76b4fc727aac482b080dcf17abfccc0e6fcf02b594ad2d0c0c13dddb1e4cf8" exitCode=60 Nov 02 23:45:16.913583 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:16.913109 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48" event={"ID":"b18f144a-0ec3-4920-b630-a34e6959fac8","Type":"ContainerDied","Data":"dd76b4fc727aac482b080dcf17abfccc0e6fcf02b594ad2d0c0c13dddb1e4cf8"} Nov 02 23:45:16.913583 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:16.913190 5114 scope.go:122] "RemoveContainer" containerID="5d0f2c2ef1a9282ba4841cf8995921219a73ba4d6096c77e563ce4858b51cc0c" Nov 02 23:45:16.913583 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:16.913556 5114 scope.go:122] "RemoveContainer" containerID="dd76b4fc727aac482b080dcf17abfccc0e6fcf02b594ad2d0c0c13dddb1e4cf8" Nov 02 23:45:16.913834 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:16.913813 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48_odh-ai-gateway-infra(b18f144a-0ec3-4920-b630-a34e6959fac8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48" podUID="b18f144a-0ec3-4920-b630-a34e6959fac8" Nov 02 23:45:16.915307 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:16.915281 5114 generic.go:356] "Generic (PLEG): container finished" podID="476aca9f-b87e-4ce3-a6d6-ea20e6bb7686" containerID="fce29d5ead521119eed43fdd6090a689643dbd4760d3ed956d818c36ebb089c9" exitCode=60 Nov 02 23:45:16.915417 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:16.915316 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl" event={"ID":"476aca9f-b87e-4ce3-a6d6-ea20e6bb7686","Type":"ContainerDied","Data":"fce29d5ead521119eed43fdd6090a689643dbd4760d3ed956d818c36ebb089c9"} Nov 02 23:45:16.915699 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:16.915681 5114 scope.go:122] "RemoveContainer" containerID="fce29d5ead521119eed43fdd6090a689643dbd4760d3ed956d818c36ebb089c9" Nov 02 23:45:16.915943 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:16.915924 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl_odh-ai-gateway-infra(476aca9f-b87e-4ce3-a6d6-ea20e6bb7686)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl" podUID="476aca9f-b87e-4ce3-a6d6-ea20e6bb7686" Nov 02 23:45:16.927312 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:16.927289 5114 scope.go:122] "RemoveContainer" containerID="91992fb3bc72d162ae94a074e1acabafc7937a47f6a0281d2a90a0b135374b89" Nov 02 23:45:17.026231 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.026144 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns"] Nov 02 23:45:17.158347 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.158316 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr"] Nov 02 23:45:17.176729 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.176693 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns" Nov 02 23:45:17.318655 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.318625 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20959fb5-d0a9-405f-bdf6-0b778ef9fc33-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20959fb5-d0a9-405f-bdf6-0b778ef9fc33-maas-api-service-ca\") pod \"20959fb5-d0a9-405f-bdf6-0b778ef9fc33\" (UID: \"20959fb5-d0a9-405f-bdf6-0b778ef9fc33\") " Nov 02 23:45:17.318833 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.318772 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20959fb5-d0a9-405f-bdf6-0b778ef9fc33-kube-api-access-hp8fv\" (UniqueName: \"kubernetes.io/projected/20959fb5-d0a9-405f-bdf6-0b778ef9fc33-kube-api-access-hp8fv\") pod \"20959fb5-d0a9-405f-bdf6-0b778ef9fc33\" (UID: \"20959fb5-d0a9-405f-bdf6-0b778ef9fc33\") " Nov 02 23:45:17.319011 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.318985 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20959fb5-d0a9-405f-bdf6-0b778ef9fc33-maas-api-service-ca" pod "20959fb5-d0a9-405f-bdf6-0b778ef9fc33" (UID: "20959fb5-d0a9-405f-bdf6-0b778ef9fc33"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:17.319176 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.319150 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20959fb5-d0a9-405f-bdf6-0b778ef9fc33-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:17.319301 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.319290 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr" Nov 02 23:45:17.321096 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.321076 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20959fb5-d0a9-405f-bdf6-0b778ef9fc33-kube-api-access-hp8fv" pod "20959fb5-d0a9-405f-bdf6-0b778ef9fc33" (UID: "20959fb5-d0a9-405f-bdf6-0b778ef9fc33"). InnerVolumeSpecName "kube-api-access-hp8fv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:17.419869 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.419830 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55787ff7-00ec-4047-a21b-71cc26ccd553-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55787ff7-00ec-4047-a21b-71cc26ccd553-maas-api-service-ca\") pod \"55787ff7-00ec-4047-a21b-71cc26ccd553\" (UID: \"55787ff7-00ec-4047-a21b-71cc26ccd553\") " Nov 02 23:45:17.419869 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.419880 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55787ff7-00ec-4047-a21b-71cc26ccd553-kube-api-access-5r8jd\" (UniqueName: \"kubernetes.io/projected/55787ff7-00ec-4047-a21b-71cc26ccd553-kube-api-access-5r8jd\") pod \"55787ff7-00ec-4047-a21b-71cc26ccd553\" (UID: \"55787ff7-00ec-4047-a21b-71cc26ccd553\") " Nov 02 23:45:17.420522 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.420457 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55787ff7-00ec-4047-a21b-71cc26ccd553-maas-api-service-ca" pod "55787ff7-00ec-4047-a21b-71cc26ccd553" (UID: "55787ff7-00ec-4047-a21b-71cc26ccd553"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:17.420783 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.420762 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp8fv\" (UniqueName: \"kubernetes.io/projected/20959fb5-d0a9-405f-bdf6-0b778ef9fc33-kube-api-access-hp8fv\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:17.420879 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.420790 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55787ff7-00ec-4047-a21b-71cc26ccd553-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:17.423111 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.423084 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55787ff7-00ec-4047-a21b-71cc26ccd553-kube-api-access-5r8jd" pod "55787ff7-00ec-4047-a21b-71cc26ccd553" (UID: "55787ff7-00ec-4047-a21b-71cc26ccd553"). InnerVolumeSpecName "kube-api-access-5r8jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:17.522290 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.522248 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r8jd\" (UniqueName: \"kubernetes.io/projected/55787ff7-00ec-4047-a21b-71cc26ccd553-kube-api-access-5r8jd\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:17.786797 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.786764 5114 scope.go:122] "RemoveContainer" containerID="d5b204337cc922820fefa20fa5cc7679a70043ba469bbd4eb345fa694975e07b" Nov 02 23:45:17.926414 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.926379 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns" Nov 02 23:45:17.926955 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.926408 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns" event={"ID":"20959fb5-d0a9-405f-bdf6-0b778ef9fc33","Type":"ContainerDied","Data":"6a8c331f471d061b1a18191c493113cf5c26ac51127f63f2412719417e859250"} Nov 02 23:45:17.926955 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.926510 5114 scope.go:122] "RemoveContainer" containerID="0c5c27ba3ea1aab34924c60b1e54d3e616af6711b9ac63fafa712b1ccb12b385" Nov 02 23:45:17.928952 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.928927 5114 generic.go:356] "Generic (PLEG): container finished" podID="33912f7f-d988-447b-b49d-b2af9b285b39" containerID="30d4775e774200dfb7be8c3b0fb9ec4087dda07c3b3d37fbf04cea47bc95a3d1" exitCode=60 Nov 02 23:45:17.929086 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.929015 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq" event={"ID":"33912f7f-d988-447b-b49d-b2af9b285b39","Type":"ContainerDied","Data":"30d4775e774200dfb7be8c3b0fb9ec4087dda07c3b3d37fbf04cea47bc95a3d1"} Nov 02 23:45:17.929464 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.929441 5114 scope.go:122] "RemoveContainer" containerID="30d4775e774200dfb7be8c3b0fb9ec4087dda07c3b3d37fbf04cea47bc95a3d1" Nov 02 23:45:17.929832 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:17.929800 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq_odh-ai-gateway-infra(33912f7f-d988-447b-b49d-b2af9b285b39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq" podUID="33912f7f-d988-447b-b49d-b2af9b285b39" Nov 02 23:45:17.934009 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.933982 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr" event={"ID":"55787ff7-00ec-4047-a21b-71cc26ccd553","Type":"ContainerDied","Data":"61e4daa7828118cbc3ca02aa8ab4470327565e486397fd75b27038a5a21ddd61"} Nov 02 23:45:17.934123 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.934014 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr" Nov 02 23:45:17.937431 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.937407 5114 scope.go:122] "RemoveContainer" containerID="45f065ce479a450abd66788ded1fc8cffdc48c430c4c104e3c6ce63082927195" Nov 02 23:45:17.950621 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:17.950597 5114 scope.go:122] "RemoveContainer" containerID="2c9e86907d32ad517ff12cf4e0d1e6bdf743259f929aed72328a416a227799f9" Nov 02 23:45:18.040288 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.040251 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48"] Nov 02 23:45:18.197811 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.197786 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48" Nov 02 23:45:18.211962 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.211926 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr"] Nov 02 23:45:18.217038 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.216970 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894385-tvczr"] Nov 02 23:45:18.305756 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.305724 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl"] Nov 02 23:45:18.331321 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.331291 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b18f144a-0ec3-4920-b630-a34e6959fac8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18f144a-0ec3-4920-b630-a34e6959fac8-maas-api-service-ca\") pod \"b18f144a-0ec3-4920-b630-a34e6959fac8\" (UID: \"b18f144a-0ec3-4920-b630-a34e6959fac8\") " Nov 02 23:45:18.331546 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.331345 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b18f144a-0ec3-4920-b630-a34e6959fac8-kube-api-access-fnjlw\" (UniqueName: \"kubernetes.io/projected/b18f144a-0ec3-4920-b630-a34e6959fac8-kube-api-access-fnjlw\") pod \"b18f144a-0ec3-4920-b630-a34e6959fac8\" (UID: \"b18f144a-0ec3-4920-b630-a34e6959fac8\") " Nov 02 23:45:18.331735 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.331712 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b18f144a-0ec3-4920-b630-a34e6959fac8-maas-api-service-ca" pod "b18f144a-0ec3-4920-b630-a34e6959fac8" (UID: "b18f144a-0ec3-4920-b630-a34e6959fac8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:18.332084 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.332045 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns"] Nov 02 23:45:18.334074 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.334048 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b18f144a-0ec3-4920-b630-a34e6959fac8-kube-api-access-fnjlw" pod "b18f144a-0ec3-4920-b630-a34e6959fac8" (UID: "b18f144a-0ec3-4920-b630-a34e6959fac8"). InnerVolumeSpecName "kube-api-access-fnjlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:18.335617 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.335583 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894385-hxlns"] Nov 02 23:45:18.433189 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.433156 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnjlw\" (UniqueName: \"kubernetes.io/projected/b18f144a-0ec3-4920-b630-a34e6959fac8-kube-api-access-fnjlw\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:18.433189 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.433187 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18f144a-0ec3-4920-b630-a34e6959fac8-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:18.453919 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.453883 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl" Nov 02 23:45:18.534218 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.534176 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/476aca9f-b87e-4ce3-a6d6-ea20e6bb7686-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/476aca9f-b87e-4ce3-a6d6-ea20e6bb7686-maas-api-service-ca\") pod \"476aca9f-b87e-4ce3-a6d6-ea20e6bb7686\" (UID: \"476aca9f-b87e-4ce3-a6d6-ea20e6bb7686\") " Nov 02 23:45:18.534428 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.534345 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/476aca9f-b87e-4ce3-a6d6-ea20e6bb7686-kube-api-access-m8hlc\" (UniqueName: \"kubernetes.io/projected/476aca9f-b87e-4ce3-a6d6-ea20e6bb7686-kube-api-access-m8hlc\") pod \"476aca9f-b87e-4ce3-a6d6-ea20e6bb7686\" (UID: \"476aca9f-b87e-4ce3-a6d6-ea20e6bb7686\") " Nov 02 23:45:18.534641 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.534613 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/476aca9f-b87e-4ce3-a6d6-ea20e6bb7686-maas-api-service-ca" pod "476aca9f-b87e-4ce3-a6d6-ea20e6bb7686" (UID: "476aca9f-b87e-4ce3-a6d6-ea20e6bb7686"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:18.534812 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.534796 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/476aca9f-b87e-4ce3-a6d6-ea20e6bb7686-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:18.536603 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.536585 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476aca9f-b87e-4ce3-a6d6-ea20e6bb7686-kube-api-access-m8hlc" pod "476aca9f-b87e-4ce3-a6d6-ea20e6bb7686" (UID: "476aca9f-b87e-4ce3-a6d6-ea20e6bb7686"). InnerVolumeSpecName "kube-api-access-m8hlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:18.636189 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.636154 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8hlc\" (UniqueName: \"kubernetes.io/projected/476aca9f-b87e-4ce3-a6d6-ea20e6bb7686-kube-api-access-m8hlc\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:18.787749 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.787715 5114 scope.go:122] "RemoveContainer" containerID="b62dc79b07699c1491e800d22a2715b9c31a1790e665d6c6770555ae961cfaee" Nov 02 23:45:18.794851 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.794823 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20959fb5-d0a9-405f-bdf6-0b778ef9fc33" path="/var/lib/kubelet/pods/20959fb5-d0a9-405f-bdf6-0b778ef9fc33/volumes" Nov 02 23:45:18.795235 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.795220 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55787ff7-00ec-4047-a21b-71cc26ccd553" path="/var/lib/kubelet/pods/55787ff7-00ec-4047-a21b-71cc26ccd553/volumes" Nov 02 23:45:18.943130 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.943045 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48" Nov 02 23:45:18.943130 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.943046 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48" event={"ID":"b18f144a-0ec3-4920-b630-a34e6959fac8","Type":"ContainerDied","Data":"07591fbd5ace4906ebd3729ca9a241bd499cad16855267308c248beb7b27faa6"} Nov 02 23:45:18.943709 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.943188 5114 scope.go:122] "RemoveContainer" containerID="dd76b4fc727aac482b080dcf17abfccc0e6fcf02b594ad2d0c0c13dddb1e4cf8" Nov 02 23:45:18.944658 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.944629 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl" Nov 02 23:45:18.944758 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.944649 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl" event={"ID":"476aca9f-b87e-4ce3-a6d6-ea20e6bb7686","Type":"ContainerDied","Data":"5674cbb475c9bfaba34ce4fce9a6b9f2a25cd53be83605ad067f5f12b40bb2cc"} Nov 02 23:45:18.949389 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.949365 5114 generic.go:356] "Generic (PLEG): container finished" podID="7598e2a8-ab19-48ca-b107-94451972d5bf" containerID="fe1ba7fe18c6a69c334c4bb7d32ed280e9a791abf20a11c9ebbd8fd7f4d01777" exitCode=60 Nov 02 23:45:18.949510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.949421 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv" event={"ID":"7598e2a8-ab19-48ca-b107-94451972d5bf","Type":"ContainerDied","Data":"fe1ba7fe18c6a69c334c4bb7d32ed280e9a791abf20a11c9ebbd8fd7f4d01777"} Nov 02 23:45:18.949816 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.949802 5114 scope.go:122] "RemoveContainer" containerID="fe1ba7fe18c6a69c334c4bb7d32ed280e9a791abf20a11c9ebbd8fd7f4d01777" Nov 02 23:45:18.950072 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:18.950054 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv_odh-ai-gateway-infra(7598e2a8-ab19-48ca-b107-94451972d5bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv" podUID="7598e2a8-ab19-48ca-b107-94451972d5bf" Nov 02 23:45:18.953173 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.953153 5114 scope.go:122] "RemoveContainer" containerID="fce29d5ead521119eed43fdd6090a689643dbd4760d3ed956d818c36ebb089c9" Nov 02 23:45:18.962167 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:18.962147 5114 scope.go:122] "RemoveContainer" containerID="d5b204337cc922820fefa20fa5cc7679a70043ba469bbd4eb345fa694975e07b" Nov 02 23:45:19.106150 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:19.106111 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq"] Nov 02 23:45:19.112417 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:19.112384 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48"] Nov 02 23:45:19.122199 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:19.122167 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-915ccb-29894385-k6f48"] Nov 02 23:45:19.231089 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:19.230986 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl"] Nov 02 23:45:19.236819 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:19.236782 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894385-smphl"] Nov 02 23:45:19.262158 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:19.262119 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq" Nov 02 23:45:19.343969 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:19.343929 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33912f7f-d988-447b-b49d-b2af9b285b39-kube-api-access-b77nm\" (UniqueName: \"kubernetes.io/projected/33912f7f-d988-447b-b49d-b2af9b285b39-kube-api-access-b77nm\") pod \"33912f7f-d988-447b-b49d-b2af9b285b39\" (UID: \"33912f7f-d988-447b-b49d-b2af9b285b39\") " Nov 02 23:45:19.344128 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:19.344060 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33912f7f-d988-447b-b49d-b2af9b285b39-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33912f7f-d988-447b-b49d-b2af9b285b39-maas-api-service-ca\") pod \"33912f7f-d988-447b-b49d-b2af9b285b39\" (UID: \"33912f7f-d988-447b-b49d-b2af9b285b39\") " Nov 02 23:45:19.344475 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:19.344451 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33912f7f-d988-447b-b49d-b2af9b285b39-maas-api-service-ca" pod "33912f7f-d988-447b-b49d-b2af9b285b39" (UID: "33912f7f-d988-447b-b49d-b2af9b285b39"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:19.344568 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:19.344547 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33912f7f-d988-447b-b49d-b2af9b285b39-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:19.346275 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:19.346242 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33912f7f-d988-447b-b49d-b2af9b285b39-kube-api-access-b77nm" pod "33912f7f-d988-447b-b49d-b2af9b285b39" (UID: "33912f7f-d988-447b-b49d-b2af9b285b39"). InnerVolumeSpecName "kube-api-access-b77nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:19.445564 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:19.445477 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b77nm\" (UniqueName: \"kubernetes.io/projected/33912f7f-d988-447b-b49d-b2af9b285b39-kube-api-access-b77nm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:19.957507 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:19.957403 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq" Nov 02 23:45:19.957507 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:19.957441 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq" event={"ID":"33912f7f-d988-447b-b49d-b2af9b285b39","Type":"ContainerDied","Data":"892be23df0e54152d3be94cc94cf48faba9ee5e01b1efa6a7b4a4c9051f73efe"} Nov 02 23:45:19.957995 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:19.957512 5114 scope.go:122] "RemoveContainer" containerID="30d4775e774200dfb7be8c3b0fb9ec4087dda07c3b3d37fbf04cea47bc95a3d1" Nov 02 23:45:19.961460 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:19.961432 5114 generic.go:356] "Generic (PLEG): container finished" podID="384a2134-0a3a-4f97-9848-02193a94df3a" containerID="55dc1674f1e4853a7c07dd4cf47c4b0402405e6d15361ae9c9fdd70d2b885682" exitCode=60 Nov 02 23:45:19.961584 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:19.961516 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w" event={"ID":"384a2134-0a3a-4f97-9848-02193a94df3a","Type":"ContainerDied","Data":"55dc1674f1e4853a7c07dd4cf47c4b0402405e6d15361ae9c9fdd70d2b885682"} Nov 02 23:45:19.961992 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:19.961973 5114 scope.go:122] "RemoveContainer" containerID="55dc1674f1e4853a7c07dd4cf47c4b0402405e6d15361ae9c9fdd70d2b885682" Nov 02 23:45:19.962319 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:19.962294 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w_odh-ai-gateway-infra(384a2134-0a3a-4f97-9848-02193a94df3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w" podUID="384a2134-0a3a-4f97-9848-02193a94df3a" Nov 02 23:45:19.968503 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:19.968458 5114 scope.go:122] "RemoveContainer" containerID="b62dc79b07699c1491e800d22a2715b9c31a1790e665d6c6770555ae961cfaee" Nov 02 23:45:20.131713 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:20.131675 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq"] Nov 02 23:45:20.137526 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:20.137469 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894385-mw9gq"] Nov 02 23:45:20.360561 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:20.360529 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv"] Nov 02 23:45:20.507545 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:20.507511 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv" Nov 02 23:45:20.657999 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:20.657917 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7598e2a8-ab19-48ca-b107-94451972d5bf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7598e2a8-ab19-48ca-b107-94451972d5bf-maas-api-service-ca\") pod \"7598e2a8-ab19-48ca-b107-94451972d5bf\" (UID: \"7598e2a8-ab19-48ca-b107-94451972d5bf\") " Nov 02 23:45:20.657999 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:20.657978 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7598e2a8-ab19-48ca-b107-94451972d5bf-kube-api-access-rg75q\" (UniqueName: \"kubernetes.io/projected/7598e2a8-ab19-48ca-b107-94451972d5bf-kube-api-access-rg75q\") pod \"7598e2a8-ab19-48ca-b107-94451972d5bf\" (UID: \"7598e2a8-ab19-48ca-b107-94451972d5bf\") " Nov 02 23:45:20.658364 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:20.658339 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7598e2a8-ab19-48ca-b107-94451972d5bf-maas-api-service-ca" pod "7598e2a8-ab19-48ca-b107-94451972d5bf" (UID: "7598e2a8-ab19-48ca-b107-94451972d5bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:20.658477 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:20.658464 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7598e2a8-ab19-48ca-b107-94451972d5bf-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:20.660308 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:20.660283 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7598e2a8-ab19-48ca-b107-94451972d5bf-kube-api-access-rg75q" pod "7598e2a8-ab19-48ca-b107-94451972d5bf" (UID: "7598e2a8-ab19-48ca-b107-94451972d5bf"). InnerVolumeSpecName "kube-api-access-rg75q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:20.759964 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:20.759920 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg75q\" (UniqueName: \"kubernetes.io/projected/7598e2a8-ab19-48ca-b107-94451972d5bf-kube-api-access-rg75q\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:20.793535 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:20.793463 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33912f7f-d988-447b-b49d-b2af9b285b39" path="/var/lib/kubelet/pods/33912f7f-d988-447b-b49d-b2af9b285b39/volumes" Nov 02 23:45:20.794097 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:20.794078 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="476aca9f-b87e-4ce3-a6d6-ea20e6bb7686" path="/var/lib/kubelet/pods/476aca9f-b87e-4ce3-a6d6-ea20e6bb7686/volumes" Nov 02 23:45:20.794512 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:20.794472 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b18f144a-0ec3-4920-b630-a34e6959fac8" path="/var/lib/kubelet/pods/b18f144a-0ec3-4920-b630-a34e6959fac8/volumes" Nov 02 23:45:20.968200 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:20.968108 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv" Nov 02 23:45:20.968200 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:20.968142 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv" event={"ID":"7598e2a8-ab19-48ca-b107-94451972d5bf","Type":"ContainerDied","Data":"d323d567428c664d3401805f1805f42c1f5e5013afecde2990d08f4cec0659ee"} Nov 02 23:45:20.968775 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:20.968219 5114 scope.go:122] "RemoveContainer" containerID="fe1ba7fe18c6a69c334c4bb7d32ed280e9a791abf20a11c9ebbd8fd7f4d01777" Nov 02 23:45:21.133650 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:21.133612 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv"] Nov 02 23:45:21.137708 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:21.137676 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f46b61-29894385-8twjv"] Nov 02 23:45:21.260707 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:21.260615 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w"] Nov 02 23:45:21.423600 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:21.423560 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w" Nov 02 23:45:21.567995 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:21.567962 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/384a2134-0a3a-4f97-9848-02193a94df3a-kube-api-access-z2ptt\" (UniqueName: \"kubernetes.io/projected/384a2134-0a3a-4f97-9848-02193a94df3a-kube-api-access-z2ptt\") pod \"384a2134-0a3a-4f97-9848-02193a94df3a\" (UID: \"384a2134-0a3a-4f97-9848-02193a94df3a\") " Nov 02 23:45:21.568220 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:21.568032 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/384a2134-0a3a-4f97-9848-02193a94df3a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/384a2134-0a3a-4f97-9848-02193a94df3a-maas-api-service-ca\") pod \"384a2134-0a3a-4f97-9848-02193a94df3a\" (UID: \"384a2134-0a3a-4f97-9848-02193a94df3a\") " Nov 02 23:45:21.568457 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:21.568427 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/384a2134-0a3a-4f97-9848-02193a94df3a-maas-api-service-ca" pod "384a2134-0a3a-4f97-9848-02193a94df3a" (UID: "384a2134-0a3a-4f97-9848-02193a94df3a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:21.570650 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:21.570612 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384a2134-0a3a-4f97-9848-02193a94df3a-kube-api-access-z2ptt" pod "384a2134-0a3a-4f97-9848-02193a94df3a" (UID: "384a2134-0a3a-4f97-9848-02193a94df3a"). InnerVolumeSpecName "kube-api-access-z2ptt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:21.669377 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:21.669333 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/384a2134-0a3a-4f97-9848-02193a94df3a-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:21.669377 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:21.669366 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2ptt\" (UniqueName: \"kubernetes.io/projected/384a2134-0a3a-4f97-9848-02193a94df3a-kube-api-access-z2ptt\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:21.980437 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:21.980342 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w" event={"ID":"384a2134-0a3a-4f97-9848-02193a94df3a","Type":"ContainerDied","Data":"09ab796bf2368d26ff00b3f6bf9402af7ca9c8b546df39fdf7fe8e3edbaa2ca2"} Nov 02 23:45:21.980437 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:21.980379 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w" Nov 02 23:45:21.980437 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:21.980403 5114 scope.go:122] "RemoveContainer" containerID="55dc1674f1e4853a7c07dd4cf47c4b0402405e6d15361ae9c9fdd70d2b885682" Nov 02 23:45:22.143435 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:22.143391 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w"] Nov 02 23:45:22.192764 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:22.192723 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894385-b6g2w"] Nov 02 23:45:22.607597 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:22.607554 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:45:22.620602 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:22.620569 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:45:22.793246 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:22.793213 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="384a2134-0a3a-4f97-9848-02193a94df3a" path="/var/lib/kubelet/pods/384a2134-0a3a-4f97-9848-02193a94df3a/volumes" Nov 02 23:45:22.793683 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:22.793666 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7598e2a8-ab19-48ca-b107-94451972d5bf" path="/var/lib/kubelet/pods/7598e2a8-ab19-48ca-b107-94451972d5bf/volumes" Nov 02 23:45:22.999662 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:22.999573 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:45:23.217605 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.217573 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7699-b0fd46"] Nov 02 23:45:23.217976 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.217963 5114 cpu_manager.go:414] "containerMap: removing container" podUID="b18f144a-0ec3-4920-b630-a34e6959fac8" containerName="cleanup" Nov 02 23:45:23.218038 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.217977 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b18f144a-0ec3-4920-b630-a34e6959fac8" containerName="cleanup" Nov 02 23:45:23.218038 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.217995 5114 cpu_manager.go:414] "containerMap: removing container" podUID="33912f7f-d988-447b-b49d-b2af9b285b39" containerName="cleanup" Nov 02 23:45:23.218038 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218001 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33912f7f-d988-447b-b49d-b2af9b285b39" containerName="cleanup" Nov 02 23:45:23.218038 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218010 5114 cpu_manager.go:414] "containerMap: removing container" podUID="476aca9f-b87e-4ce3-a6d6-ea20e6bb7686" containerName="cleanup" Nov 02 23:45:23.218038 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218015 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="476aca9f-b87e-4ce3-a6d6-ea20e6bb7686" containerName="cleanup" Nov 02 23:45:23.218038 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218021 5114 cpu_manager.go:414] "containerMap: removing container" podUID="384a2134-0a3a-4f97-9848-02193a94df3a" containerName="cleanup" Nov 02 23:45:23.218038 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218026 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="384a2134-0a3a-4f97-9848-02193a94df3a" containerName="cleanup" Nov 02 23:45:23.218038 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218034 5114 cpu_manager.go:414] "containerMap: removing container" podUID="b18f144a-0ec3-4920-b630-a34e6959fac8" containerName="cleanup" Nov 02 23:45:23.218038 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218038 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b18f144a-0ec3-4920-b630-a34e6959fac8" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218046 5114 cpu_manager.go:414] "containerMap: removing container" podUID="7598e2a8-ab19-48ca-b107-94451972d5bf" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218051 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7598e2a8-ab19-48ca-b107-94451972d5bf" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218060 5114 cpu_manager.go:414] "containerMap: removing container" podUID="7598e2a8-ab19-48ca-b107-94451972d5bf" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218064 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7598e2a8-ab19-48ca-b107-94451972d5bf" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218072 5114 cpu_manager.go:414] "containerMap: removing container" podUID="7598e2a8-ab19-48ca-b107-94451972d5bf" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218078 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7598e2a8-ab19-48ca-b107-94451972d5bf" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218087 5114 cpu_manager.go:414] "containerMap: removing container" podUID="55787ff7-00ec-4047-a21b-71cc26ccd553" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218091 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55787ff7-00ec-4047-a21b-71cc26ccd553" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218098 5114 cpu_manager.go:414] "containerMap: removing container" podUID="33912f7f-d988-447b-b49d-b2af9b285b39" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218102 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33912f7f-d988-447b-b49d-b2af9b285b39" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218117 5114 cpu_manager.go:414] "containerMap: removing container" podUID="20959fb5-d0a9-405f-bdf6-0b778ef9fc33" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218121 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20959fb5-d0a9-405f-bdf6-0b778ef9fc33" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218130 5114 cpu_manager.go:414] "containerMap: removing container" podUID="20959fb5-d0a9-405f-bdf6-0b778ef9fc33" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218135 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20959fb5-d0a9-405f-bdf6-0b778ef9fc33" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218143 5114 cpu_manager.go:414] "containerMap: removing container" podUID="476aca9f-b87e-4ce3-a6d6-ea20e6bb7686" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218147 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="476aca9f-b87e-4ce3-a6d6-ea20e6bb7686" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218154 5114 cpu_manager.go:414] "containerMap: removing container" podUID="384a2134-0a3a-4f97-9848-02193a94df3a" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218158 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="384a2134-0a3a-4f97-9848-02193a94df3a" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218164 5114 cpu_manager.go:414] "containerMap: removing container" podUID="55787ff7-00ec-4047-a21b-71cc26ccd553" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218168 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55787ff7-00ec-4047-a21b-71cc26ccd553" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218222 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="55787ff7-00ec-4047-a21b-71cc26ccd553" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218229 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="33912f7f-d988-447b-b49d-b2af9b285b39" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218235 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="476aca9f-b87e-4ce3-a6d6-ea20e6bb7686" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218242 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="20959fb5-d0a9-405f-bdf6-0b778ef9fc33" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218247 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="384a2134-0a3a-4f97-9848-02193a94df3a" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218252 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="20959fb5-d0a9-405f-bdf6-0b778ef9fc33" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218257 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="476aca9f-b87e-4ce3-a6d6-ea20e6bb7686" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218263 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="55787ff7-00ec-4047-a21b-71cc26ccd553" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218270 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="384a2134-0a3a-4f97-9848-02193a94df3a" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218275 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="20959fb5-d0a9-405f-bdf6-0b778ef9fc33" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218280 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="b18f144a-0ec3-4920-b630-a34e6959fac8" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218286 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="55787ff7-00ec-4047-a21b-71cc26ccd553" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218290 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="b18f144a-0ec3-4920-b630-a34e6959fac8" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218296 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="7598e2a8-ab19-48ca-b107-94451972d5bf" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218301 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="33912f7f-d988-447b-b49d-b2af9b285b39" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218305 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="b18f144a-0ec3-4920-b630-a34e6959fac8" containerName="cleanup" Nov 02 23:45:23.218333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.218311 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="7598e2a8-ab19-48ca-b107-94451972d5bf" containerName="cleanup" Nov 02 23:45:23.222920 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.222891 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-b0fd46" Nov 02 23:45:23.275531 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.275399 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-b8v4g\"" Nov 02 23:45:23.277324 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.277293 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-b0fd46"] Nov 02 23:45:23.390109 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.390064 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtt7d\" (UniqueName: \"kubernetes.io/projected/ba63dded-aebb-482c-9f1b-d21ae07b124b-kube-api-access-rtt7d\") pod \"test-curl-7699-b0fd46\" (UID: \"ba63dded-aebb-482c-9f1b-d21ae07b124b\") " pod="opendatahub/test-curl-7699-b0fd46" Nov 02 23:45:23.491261 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.491222 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtt7d\" (UniqueName: \"kubernetes.io/projected/ba63dded-aebb-482c-9f1b-d21ae07b124b-kube-api-access-rtt7d\") pod \"test-curl-7699-b0fd46\" (UID: \"ba63dded-aebb-482c-9f1b-d21ae07b124b\") " pod="opendatahub/test-curl-7699-b0fd46" Nov 02 23:45:23.552394 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.552368 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtt7d\" (UniqueName: \"kubernetes.io/projected/ba63dded-aebb-482c-9f1b-d21ae07b124b-kube-api-access-rtt7d\") pod \"test-curl-7699-b0fd46\" (UID: \"ba63dded-aebb-482c-9f1b-d21ae07b124b\") " pod="opendatahub/test-curl-7699-b0fd46" Nov 02 23:45:23.836140 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:23.836051 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-b0fd46" Nov 02 23:45:24.151050 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:24.150949 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" podUID="737c5cf0-065a-4223-97be-3176aebab7d6" containerName="main" containerID="cri-o://c3989486acdf4af45a0d698e0e268c1ab4aa0e8d8df4fd08f3134a7a8594fa05" gracePeriod=45 Nov 02 23:45:24.209093 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:24.209067 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-b0fd46"] Nov 02 23:45:24.230651 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:45:24.230612 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba63dded_aebb_482c_9f1b_d21ae07b124b.slice/crio-64c8d812776cc761a65f029b88887c2b1a75cc63347f5eede42fb79d9d8014f7 WatchSource:0}: Error finding container 64c8d812776cc761a65f029b88887c2b1a75cc63347f5eede42fb79d9d8014f7: Status 404 returned error can't find the container with id 64c8d812776cc761a65f029b88887c2b1a75cc63347f5eede42fb79d9d8014f7 Nov 02 23:45:24.392210 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:24.392184 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:45:24.510881 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:24.510787 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-model-cache\") pod \"737c5cf0-065a-4223-97be-3176aebab7d6\" (UID: \"737c5cf0-065a-4223-97be-3176aebab7d6\") " Nov 02 23:45:24.510881 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:24.510857 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/737c5cf0-065a-4223-97be-3176aebab7d6-kube-api-access-w6nf9\" (UniqueName: \"kubernetes.io/projected/737c5cf0-065a-4223-97be-3176aebab7d6-kube-api-access-w6nf9\") pod \"737c5cf0-065a-4223-97be-3176aebab7d6\" (UID: \"737c5cf0-065a-4223-97be-3176aebab7d6\") " Nov 02 23:45:24.511096 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:24.510912 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-home\" (UniqueName: \"kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-home\") pod \"737c5cf0-065a-4223-97be-3176aebab7d6\" (UID: \"737c5cf0-065a-4223-97be-3176aebab7d6\") " Nov 02 23:45:24.511096 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:24.510970 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-tmp-dir\") pod \"737c5cf0-065a-4223-97be-3176aebab7d6\" (UID: \"737c5cf0-065a-4223-97be-3176aebab7d6\") " Nov 02 23:45:24.511096 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:24.511015 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-dshm\") pod \"737c5cf0-065a-4223-97be-3176aebab7d6\" (UID: \"737c5cf0-065a-4223-97be-3176aebab7d6\") " Nov 02 23:45:24.511096 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:24.511084 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/737c5cf0-065a-4223-97be-3176aebab7d6-tls-certs\" (UniqueName: \"kubernetes.io/secret/737c5cf0-065a-4223-97be-3176aebab7d6-tls-certs\") pod \"737c5cf0-065a-4223-97be-3176aebab7d6\" (UID: \"737c5cf0-065a-4223-97be-3176aebab7d6\") " Nov 02 23:45:24.511318 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:24.511090 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-model-cache" pod "737c5cf0-065a-4223-97be-3176aebab7d6" (UID: "737c5cf0-065a-4223-97be-3176aebab7d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:24.511318 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:24.511188 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-home" pod "737c5cf0-065a-4223-97be-3176aebab7d6" (UID: "737c5cf0-065a-4223-97be-3176aebab7d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:24.511318 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:24.511222 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-tmp-dir" pod "737c5cf0-065a-4223-97be-3176aebab7d6" (UID: "737c5cf0-065a-4223-97be-3176aebab7d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:24.511580 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:24.511556 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:24.511580 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:24.511580 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:24.511949 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:24.511593 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:24.513550 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:24.513518 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/737c5cf0-065a-4223-97be-3176aebab7d6-tls-certs" pod "737c5cf0-065a-4223-97be-3176aebab7d6" (UID: "737c5cf0-065a-4223-97be-3176aebab7d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:45:24.513550 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:24.513539 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-dshm" pod "737c5cf0-065a-4223-97be-3176aebab7d6" (UID: "737c5cf0-065a-4223-97be-3176aebab7d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:24.513698 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:24.513622 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/737c5cf0-065a-4223-97be-3176aebab7d6-kube-api-access-w6nf9" pod "737c5cf0-065a-4223-97be-3176aebab7d6" (UID: "737c5cf0-065a-4223-97be-3176aebab7d6"). InnerVolumeSpecName "kube-api-access-w6nf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:24.612470 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:24.612428 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6nf9\" (UniqueName: \"kubernetes.io/projected/737c5cf0-065a-4223-97be-3176aebab7d6-kube-api-access-w6nf9\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:24.612470 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:24.612461 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/737c5cf0-065a-4223-97be-3176aebab7d6-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:24.612470 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:24.612472 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/737c5cf0-065a-4223-97be-3176aebab7d6-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:25.002392 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:25.002355 5114 generic.go:356] "Generic (PLEG): container finished" podID="737c5cf0-065a-4223-97be-3176aebab7d6" containerID="c3989486acdf4af45a0d698e0e268c1ab4aa0e8d8df4fd08f3134a7a8594fa05" exitCode=0 Nov 02 23:45:25.002591 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:25.002401 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" Nov 02 23:45:25.002591 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:25.002438 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" event={"ID":"737c5cf0-065a-4223-97be-3176aebab7d6","Type":"ContainerDied","Data":"c3989486acdf4af45a0d698e0e268c1ab4aa0e8d8df4fd08f3134a7a8594fa05"} Nov 02 23:45:25.002591 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:25.002506 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz" event={"ID":"737c5cf0-065a-4223-97be-3176aebab7d6","Type":"ContainerDied","Data":"323adfb9d7e2bc57ad3f18f5d1923f28a9dfca4bb65961958fbd5643b2d45660"} Nov 02 23:45:25.002591 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:25.002527 5114 scope.go:122] "RemoveContainer" containerID="c3989486acdf4af45a0d698e0e268c1ab4aa0e8d8df4fd08f3134a7a8594fa05" Nov 02 23:45:25.004030 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:25.004007 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-b0fd46" event={"ID":"ba63dded-aebb-482c-9f1b-d21ae07b124b","Type":"ContainerStarted","Data":"64c8d812776cc761a65f029b88887c2b1a75cc63347f5eede42fb79d9d8014f7"} Nov 02 23:45:25.012661 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:25.012637 5114 scope.go:122] "RemoveContainer" containerID="c3989486acdf4af45a0d698e0e268c1ab4aa0e8d8df4fd08f3134a7a8594fa05" Nov 02 23:45:25.012988 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:25.012966 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3989486acdf4af45a0d698e0e268c1ab4aa0e8d8df4fd08f3134a7a8594fa05\": container with ID starting with c3989486acdf4af45a0d698e0e268c1ab4aa0e8d8df4fd08f3134a7a8594fa05 not found: ID does not exist" containerID="c3989486acdf4af45a0d698e0e268c1ab4aa0e8d8df4fd08f3134a7a8594fa05" Nov 02 23:45:25.013069 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:25.013004 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3989486acdf4af45a0d698e0e268c1ab4aa0e8d8df4fd08f3134a7a8594fa05"} err="failed to get container status \"c3989486acdf4af45a0d698e0e268c1ab4aa0e8d8df4fd08f3134a7a8594fa05\": rpc error: code = NotFound desc = could not find container \"c3989486acdf4af45a0d698e0e268c1ab4aa0e8d8df4fd08f3134a7a8594fa05\": container with ID starting with c3989486acdf4af45a0d698e0e268c1ab4aa0e8d8df4fd08f3134a7a8594fa05 not found: ID does not exist" Nov 02 23:45:25.165891 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:25.165804 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz"] Nov 02 23:45:25.215508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:25.215437 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1f0009/auth-test-model-1f0009-kserve-7686cff4f7-xd4xz"] Nov 02 23:45:26.012568 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:26.012526 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-b0fd46" event={"ID":"ba63dded-aebb-482c-9f1b-d21ae07b124b","Type":"ContainerStarted","Data":"bbe4b5e4c2aa36a49036cc91f91a39646443ca54067e7913f310048da15f5e5a"} Nov 02 23:45:26.125356 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:26.125292 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7699-b0fd46" podStartSLOduration=2.036624134 podStartE2EDuration="3.125277131s" totalImagesPullingTime="1.088652997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:26.124638971 +0000 UTC m=+3401.477672346" watchObservedRunningTime="2026-11-02 23:45:26.125277131 +0000 UTC m=+3401.478310505" Nov 02 23:45:26.794457 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:26.794423 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="737c5cf0-065a-4223-97be-3176aebab7d6" path="/var/lib/kubelet/pods/737c5cf0-065a-4223-97be-3176aebab7d6/volumes" Nov 02 23:45:27.680629 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:27.680593 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7699-b0fd46"] Nov 02 23:45:27.685362 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:27.685327 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7699-b0fd46"] Nov 02 23:45:28.027734 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.027632 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7699-b0fd46" podUID="ba63dded-aebb-482c-9f1b-d21ae07b124b" containerName="test-curl-7699-b0fd46" containerID="cri-o://bbe4b5e4c2aa36a49036cc91f91a39646443ca54067e7913f310048da15f5e5a" gracePeriod=2 Nov 02 23:45:28.061353 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.061303 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7699-751373"] Nov 02 23:45:28.061794 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.061772 5114 cpu_manager.go:414] "containerMap: removing container" podUID="ba63dded-aebb-482c-9f1b-d21ae07b124b" containerName="test-curl-7699-b0fd46" Nov 02 23:45:28.061794 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.061793 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba63dded-aebb-482c-9f1b-d21ae07b124b" containerName="test-curl-7699-b0fd46" Nov 02 23:45:28.061975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.061805 5114 cpu_manager.go:414] "containerMap: removing container" podUID="20959fb5-d0a9-405f-bdf6-0b778ef9fc33" containerName="cleanup" Nov 02 23:45:28.061975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.061811 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20959fb5-d0a9-405f-bdf6-0b778ef9fc33" containerName="cleanup" Nov 02 23:45:28.061975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.061826 5114 cpu_manager.go:414] "containerMap: removing container" podUID="384a2134-0a3a-4f97-9848-02193a94df3a" containerName="cleanup" Nov 02 23:45:28.061975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.061836 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="384a2134-0a3a-4f97-9848-02193a94df3a" containerName="cleanup" Nov 02 23:45:28.061975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.061848 5114 cpu_manager.go:414] "containerMap: removing container" podUID="737c5cf0-065a-4223-97be-3176aebab7d6" containerName="main" Nov 02 23:45:28.061975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.061853 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="737c5cf0-065a-4223-97be-3176aebab7d6" containerName="main" Nov 02 23:45:28.061975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.061865 5114 cpu_manager.go:414] "containerMap: removing container" podUID="476aca9f-b87e-4ce3-a6d6-ea20e6bb7686" containerName="cleanup" Nov 02 23:45:28.061975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.061870 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="476aca9f-b87e-4ce3-a6d6-ea20e6bb7686" containerName="cleanup" Nov 02 23:45:28.061975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.061876 5114 cpu_manager.go:414] "containerMap: removing container" podUID="b18f144a-0ec3-4920-b630-a34e6959fac8" containerName="cleanup" Nov 02 23:45:28.061975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.061881 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b18f144a-0ec3-4920-b630-a34e6959fac8" containerName="cleanup" Nov 02 23:45:28.061975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.061894 5114 cpu_manager.go:414] "containerMap: removing container" podUID="55787ff7-00ec-4047-a21b-71cc26ccd553" containerName="cleanup" Nov 02 23:45:28.061975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.061899 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55787ff7-00ec-4047-a21b-71cc26ccd553" containerName="cleanup" Nov 02 23:45:28.061975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.061905 5114 cpu_manager.go:414] "containerMap: removing container" podUID="33912f7f-d988-447b-b49d-b2af9b285b39" containerName="cleanup" Nov 02 23:45:28.061975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.061909 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33912f7f-d988-447b-b49d-b2af9b285b39" containerName="cleanup" Nov 02 23:45:28.062508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.061983 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="476aca9f-b87e-4ce3-a6d6-ea20e6bb7686" containerName="cleanup" Nov 02 23:45:28.062508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.061994 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="384a2134-0a3a-4f97-9848-02193a94df3a" containerName="cleanup" Nov 02 23:45:28.062508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.061999 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba63dded-aebb-482c-9f1b-d21ae07b124b" containerName="test-curl-7699-b0fd46" Nov 02 23:45:28.062508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.062007 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="737c5cf0-065a-4223-97be-3176aebab7d6" containerName="main" Nov 02 23:45:28.062508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.062014 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="7598e2a8-ab19-48ca-b107-94451972d5bf" containerName="cleanup" Nov 02 23:45:28.062508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.062021 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="33912f7f-d988-447b-b49d-b2af9b285b39" containerName="cleanup" Nov 02 23:45:28.065504 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.065465 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-751373" Nov 02 23:45:28.078790 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:28.078751 5114 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7699-b0fd46\" is forbidden: User \"system:node:ip-10-0-135-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-250.us-west-1.compute.internal' and this object" podUID="ba63dded-aebb-482c-9f1b-d21ae07b124b" pod="opendatahub/test-curl-7699-b0fd46" Nov 02 23:45:28.119752 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.119703 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-751373"] Nov 02 23:45:28.147388 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.147337 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjqjk\" (UniqueName: \"kubernetes.io/projected/33d78da4-c7a8-4be3-a0d2-79539b2fe36d-kube-api-access-kjqjk\") pod \"test-curl-7699-751373\" (UID: \"33d78da4-c7a8-4be3-a0d2-79539b2fe36d\") " pod="opendatahub/test-curl-7699-751373" Nov 02 23:45:28.240519 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:28.240448 5114 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7699-b0fd46\" is forbidden: User \"system:node:ip-10-0-135-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-250.us-west-1.compute.internal' and this object" podUID="ba63dded-aebb-482c-9f1b-d21ae07b124b" pod="opendatahub/test-curl-7699-b0fd46" Nov 02 23:45:28.248540 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.248505 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjqjk\" (UniqueName: \"kubernetes.io/projected/33d78da4-c7a8-4be3-a0d2-79539b2fe36d-kube-api-access-kjqjk\") pod \"test-curl-7699-751373\" (UID: \"33d78da4-c7a8-4be3-a0d2-79539b2fe36d\") " pod="opendatahub/test-curl-7699-751373" Nov 02 23:45:28.309051 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.309022 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjqjk\" (UniqueName: \"kubernetes.io/projected/33d78da4-c7a8-4be3-a0d2-79539b2fe36d-kube-api-access-kjqjk\") pod \"test-curl-7699-751373\" (UID: \"33d78da4-c7a8-4be3-a0d2-79539b2fe36d\") " pod="opendatahub/test-curl-7699-751373" Nov 02 23:45:28.377575 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.377539 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-751373" Nov 02 23:45:28.757343 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:28.757315 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-751373"] Nov 02 23:45:28.759025 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:45:28.758988 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33d78da4_c7a8_4be3_a0d2_79539b2fe36d.slice/crio-e526566fc83b06ca4c135f54d82abe669c3d74314d9948d460b6e84ff08410eb WatchSource:0}: Error finding container e526566fc83b06ca4c135f54d82abe669c3d74314d9948d460b6e84ff08410eb: Status 404 returned error can't find the container with id e526566fc83b06ca4c135f54d82abe669c3d74314d9948d460b6e84ff08410eb Nov 02 23:45:29.035424 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:29.035389 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-751373" event={"ID":"33d78da4-c7a8-4be3-a0d2-79539b2fe36d","Type":"ContainerStarted","Data":"4e4f1294e8b34efb6d3d2ed5aab7fd81e5bae182a690dfdaa0d4b2dffb43e63c"} Nov 02 23:45:29.035899 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:29.035445 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-751373" event={"ID":"33d78da4-c7a8-4be3-a0d2-79539b2fe36d","Type":"ContainerStarted","Data":"e526566fc83b06ca4c135f54d82abe669c3d74314d9948d460b6e84ff08410eb"} Nov 02 23:45:29.149387 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:29.148444 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7699-751373" podStartSLOduration=1.1484245610000001 podStartE2EDuration="1.148424561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:29.14777593 +0000 UTC m=+3404.500809304" watchObservedRunningTime="2026-11-02 23:45:29.148424561 +0000 UTC m=+3404.501457937" Nov 02 23:45:30.267607 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:30.267581 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-b0fd46" Nov 02 23:45:30.373302 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:30.373267 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba63dded-aebb-482c-9f1b-d21ae07b124b-kube-api-access-rtt7d\" (UniqueName: \"kubernetes.io/projected/ba63dded-aebb-482c-9f1b-d21ae07b124b-kube-api-access-rtt7d\") pod \"ba63dded-aebb-482c-9f1b-d21ae07b124b\" (UID: \"ba63dded-aebb-482c-9f1b-d21ae07b124b\") " Nov 02 23:45:30.375777 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:30.375747 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba63dded-aebb-482c-9f1b-d21ae07b124b-kube-api-access-rtt7d" pod "ba63dded-aebb-482c-9f1b-d21ae07b124b" (UID: "ba63dded-aebb-482c-9f1b-d21ae07b124b"). InnerVolumeSpecName "kube-api-access-rtt7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:30.474831 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:30.474787 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtt7d\" (UniqueName: \"kubernetes.io/projected/ba63dded-aebb-482c-9f1b-d21ae07b124b-kube-api-access-rtt7d\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:30.541290 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:30.541249 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7699-751373"] Nov 02 23:45:30.545186 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:30.545153 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7699-751373"] Nov 02 23:45:30.682625 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:30.682545 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7699-22c384"] Nov 02 23:45:30.682975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:30.682960 5114 cpu_manager.go:414] "containerMap: removing container" podUID="33d78da4-c7a8-4be3-a0d2-79539b2fe36d" containerName="test-curl-7699-751373" Nov 02 23:45:30.683038 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:30.682978 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33d78da4-c7a8-4be3-a0d2-79539b2fe36d" containerName="test-curl-7699-751373" Nov 02 23:45:30.683075 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:30.683062 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="33d78da4-c7a8-4be3-a0d2-79539b2fe36d" containerName="test-curl-7699-751373" Nov 02 23:45:30.686288 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:30.686269 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-22c384" Nov 02 23:45:30.740075 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:30.740034 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-22c384"] Nov 02 23:45:30.777905 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:30.777868 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cxc4\" (UniqueName: \"kubernetes.io/projected/4811f132-cb2e-461b-8e9b-522eb00ecad9-kube-api-access-2cxc4\") pod \"test-curl-7699-22c384\" (UID: \"4811f132-cb2e-461b-8e9b-522eb00ecad9\") " pod="opendatahub/test-curl-7699-22c384" Nov 02 23:45:30.793286 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:30.793247 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba63dded-aebb-482c-9f1b-d21ae07b124b" path="/var/lib/kubelet/pods/ba63dded-aebb-482c-9f1b-d21ae07b124b/volumes" Nov 02 23:45:30.878719 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:30.878685 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cxc4\" (UniqueName: \"kubernetes.io/projected/4811f132-cb2e-461b-8e9b-522eb00ecad9-kube-api-access-2cxc4\") pod \"test-curl-7699-22c384\" (UID: \"4811f132-cb2e-461b-8e9b-522eb00ecad9\") " pod="opendatahub/test-curl-7699-22c384" Nov 02 23:45:30.940414 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:30.940338 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cxc4\" (UniqueName: \"kubernetes.io/projected/4811f132-cb2e-461b-8e9b-522eb00ecad9-kube-api-access-2cxc4\") pod \"test-curl-7699-22c384\" (UID: \"4811f132-cb2e-461b-8e9b-522eb00ecad9\") " pod="opendatahub/test-curl-7699-22c384" Nov 02 23:45:30.999695 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:30.999659 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-22c384" Nov 02 23:45:31.069529 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:31.065207 5114 generic.go:356] "Generic (PLEG): container finished" podID="ba63dded-aebb-482c-9f1b-d21ae07b124b" containerID="bbe4b5e4c2aa36a49036cc91f91a39646443ca54067e7913f310048da15f5e5a" exitCode=137 Nov 02 23:45:31.069529 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:31.065260 5114 scope.go:122] "RemoveContainer" containerID="bbe4b5e4c2aa36a49036cc91f91a39646443ca54067e7913f310048da15f5e5a" Nov 02 23:45:31.069529 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:31.065258 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-b0fd46" Nov 02 23:45:31.069529 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:31.065361 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7699-751373" podUID="33d78da4-c7a8-4be3-a0d2-79539b2fe36d" containerName="test-curl-7699-751373" containerID="cri-o://4e4f1294e8b34efb6d3d2ed5aab7fd81e5bae182a690dfdaa0d4b2dffb43e63c" gracePeriod=2 Nov 02 23:45:31.079050 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:31.078923 5114 scope.go:122] "RemoveContainer" containerID="bbe4b5e4c2aa36a49036cc91f91a39646443ca54067e7913f310048da15f5e5a" Nov 02 23:45:31.079407 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:31.079376 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbe4b5e4c2aa36a49036cc91f91a39646443ca54067e7913f310048da15f5e5a\": container with ID starting with bbe4b5e4c2aa36a49036cc91f91a39646443ca54067e7913f310048da15f5e5a not found: ID does not exist" containerID="bbe4b5e4c2aa36a49036cc91f91a39646443ca54067e7913f310048da15f5e5a" Nov 02 23:45:31.079521 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:31.079427 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbe4b5e4c2aa36a49036cc91f91a39646443ca54067e7913f310048da15f5e5a"} err="failed to get container status \"bbe4b5e4c2aa36a49036cc91f91a39646443ca54067e7913f310048da15f5e5a\": rpc error: code = NotFound desc = could not find container \"bbe4b5e4c2aa36a49036cc91f91a39646443ca54067e7913f310048da15f5e5a\": container with ID starting with bbe4b5e4c2aa36a49036cc91f91a39646443ca54067e7913f310048da15f5e5a not found: ID does not exist" Nov 02 23:45:31.376101 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:31.376068 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-22c384"] Nov 02 23:45:31.377986 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:45:31.377938 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4811f132_cb2e_461b_8e9b_522eb00ecad9.slice/crio-966c398ede9fd3766580a65007e60e047029b09f0c1be22220a49e0a01029646 WatchSource:0}: Error finding container 966c398ede9fd3766580a65007e60e047029b09f0c1be22220a49e0a01029646: Status 404 returned error can't find the container with id 966c398ede9fd3766580a65007e60e047029b09f0c1be22220a49e0a01029646 Nov 02 23:45:32.072589 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:32.072543 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-22c384" event={"ID":"4811f132-cb2e-461b-8e9b-522eb00ecad9","Type":"ContainerStarted","Data":"9a1eb82c5cd549936a2fed2eb41b6fd40dab1c897d015a94a692f1b8ee95f43f"} Nov 02 23:45:32.072822 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:32.072601 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-22c384" event={"ID":"4811f132-cb2e-461b-8e9b-522eb00ecad9","Type":"ContainerStarted","Data":"966c398ede9fd3766580a65007e60e047029b09f0c1be22220a49e0a01029646"} Nov 02 23:45:32.190125 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:32.190058 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7699-22c384" podStartSLOduration=2.190043381 podStartE2EDuration="2.190043381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:32.188270603 +0000 UTC m=+3407.541303979" watchObservedRunningTime="2026-11-02 23:45:32.190043381 +0000 UTC m=+3407.543076755" Nov 02 23:45:33.327387 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:33.327317 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-751373" Nov 02 23:45:33.408062 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:33.408028 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33d78da4-c7a8-4be3-a0d2-79539b2fe36d-kube-api-access-kjqjk\" (UniqueName: \"kubernetes.io/projected/33d78da4-c7a8-4be3-a0d2-79539b2fe36d-kube-api-access-kjqjk\") pod \"33d78da4-c7a8-4be3-a0d2-79539b2fe36d\" (UID: \"33d78da4-c7a8-4be3-a0d2-79539b2fe36d\") " Nov 02 23:45:33.410632 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:33.410598 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d78da4-c7a8-4be3-a0d2-79539b2fe36d-kube-api-access-kjqjk" pod "33d78da4-c7a8-4be3-a0d2-79539b2fe36d" (UID: "33d78da4-c7a8-4be3-a0d2-79539b2fe36d"). InnerVolumeSpecName "kube-api-access-kjqjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:33.509356 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:33.509315 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjqjk\" (UniqueName: \"kubernetes.io/projected/33d78da4-c7a8-4be3-a0d2-79539b2fe36d-kube-api-access-kjqjk\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:33.662748 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:33.662640 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7699-22c384"] Nov 02 23:45:33.668254 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:33.668215 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7699-22c384"] Nov 02 23:45:33.809277 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:33.809240 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7699-fef6cd"] Nov 02 23:45:33.809682 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:33.809667 5114 cpu_manager.go:414] "containerMap: removing container" podUID="4811f132-cb2e-461b-8e9b-522eb00ecad9" containerName="test-curl-7699-22c384" Nov 02 23:45:33.809733 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:33.809683 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4811f132-cb2e-461b-8e9b-522eb00ecad9" containerName="test-curl-7699-22c384" Nov 02 23:45:33.809813 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:33.809802 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="4811f132-cb2e-461b-8e9b-522eb00ecad9" containerName="test-curl-7699-22c384" Nov 02 23:45:33.813031 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:33.813008 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-fef6cd" Nov 02 23:45:33.869401 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:33.869371 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-fef6cd"] Nov 02 23:45:33.914559 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:33.914445 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq6m8\" (UniqueName: \"kubernetes.io/projected/a1061a95-f25f-4e3c-8e61-af12d8d717b8-kube-api-access-pq6m8\") pod \"test-curl-7699-fef6cd\" (UID: \"a1061a95-f25f-4e3c-8e61-af12d8d717b8\") " pod="opendatahub/test-curl-7699-fef6cd" Nov 02 23:45:34.015279 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:34.015238 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq6m8\" (UniqueName: \"kubernetes.io/projected/a1061a95-f25f-4e3c-8e61-af12d8d717b8-kube-api-access-pq6m8\") pod \"test-curl-7699-fef6cd\" (UID: \"a1061a95-f25f-4e3c-8e61-af12d8d717b8\") " pod="opendatahub/test-curl-7699-fef6cd" Nov 02 23:45:34.076130 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:34.076093 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq6m8\" (UniqueName: \"kubernetes.io/projected/a1061a95-f25f-4e3c-8e61-af12d8d717b8-kube-api-access-pq6m8\") pod \"test-curl-7699-fef6cd\" (UID: \"a1061a95-f25f-4e3c-8e61-af12d8d717b8\") " pod="opendatahub/test-curl-7699-fef6cd" Nov 02 23:45:34.087435 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:34.087399 5114 generic.go:356] "Generic (PLEG): container finished" podID="33d78da4-c7a8-4be3-a0d2-79539b2fe36d" containerID="4e4f1294e8b34efb6d3d2ed5aab7fd81e5bae182a690dfdaa0d4b2dffb43e63c" exitCode=137 Nov 02 23:45:34.087589 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:34.087447 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-751373" Nov 02 23:45:34.087589 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:34.087540 5114 scope.go:122] "RemoveContainer" containerID="4e4f1294e8b34efb6d3d2ed5aab7fd81e5bae182a690dfdaa0d4b2dffb43e63c" Nov 02 23:45:34.087825 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:34.087798 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7699-22c384" podUID="4811f132-cb2e-461b-8e9b-522eb00ecad9" containerName="test-curl-7699-22c384" containerID="cri-o://9a1eb82c5cd549936a2fed2eb41b6fd40dab1c897d015a94a692f1b8ee95f43f" gracePeriod=2 Nov 02 23:45:34.097124 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:34.097106 5114 scope.go:122] "RemoveContainer" containerID="4e4f1294e8b34efb6d3d2ed5aab7fd81e5bae182a690dfdaa0d4b2dffb43e63c" Nov 02 23:45:34.097471 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:34.097446 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e4f1294e8b34efb6d3d2ed5aab7fd81e5bae182a690dfdaa0d4b2dffb43e63c\": container with ID starting with 4e4f1294e8b34efb6d3d2ed5aab7fd81e5bae182a690dfdaa0d4b2dffb43e63c not found: ID does not exist" containerID="4e4f1294e8b34efb6d3d2ed5aab7fd81e5bae182a690dfdaa0d4b2dffb43e63c" Nov 02 23:45:34.097608 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:34.097508 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e4f1294e8b34efb6d3d2ed5aab7fd81e5bae182a690dfdaa0d4b2dffb43e63c"} err="failed to get container status \"4e4f1294e8b34efb6d3d2ed5aab7fd81e5bae182a690dfdaa0d4b2dffb43e63c\": rpc error: code = NotFound desc = could not find container \"4e4f1294e8b34efb6d3d2ed5aab7fd81e5bae182a690dfdaa0d4b2dffb43e63c\": container with ID starting with 4e4f1294e8b34efb6d3d2ed5aab7fd81e5bae182a690dfdaa0d4b2dffb43e63c not found: ID does not exist" Nov 02 23:45:34.126107 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:34.126075 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-fef6cd" Nov 02 23:45:34.138578 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:34.138544 5114 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7699-22c384\" is forbidden: User \"system:node:ip-10-0-135-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-250.us-west-1.compute.internal' and this object" podUID="4811f132-cb2e-461b-8e9b-522eb00ecad9" pod="opendatahub/test-curl-7699-22c384" Nov 02 23:45:34.191233 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:34.191198 5114 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7699-22c384\" is forbidden: User \"system:node:ip-10-0-135-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-250.us-west-1.compute.internal' and this object" podUID="4811f132-cb2e-461b-8e9b-522eb00ecad9" pod="opendatahub/test-curl-7699-22c384" Nov 02 23:45:34.241931 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:34.241882 5114 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7699-751373\" is forbidden: User \"system:node:ip-10-0-135-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-250.us-west-1.compute.internal' and this object" podUID="33d78da4-c7a8-4be3-a0d2-79539b2fe36d" pod="opendatahub/test-curl-7699-751373" Nov 02 23:45:34.500387 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:34.500359 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-fef6cd"] Nov 02 23:45:34.502622 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:45:34.502589 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1061a95_f25f_4e3c_8e61_af12d8d717b8.slice/crio-fd71762c3146f135441430015063f716419ad081dd804f1c227059914d180753 WatchSource:0}: Error finding container fd71762c3146f135441430015063f716419ad081dd804f1c227059914d180753: Status 404 returned error can't find the container with id fd71762c3146f135441430015063f716419ad081dd804f1c227059914d180753 Nov 02 23:45:34.793826 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:34.793787 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33d78da4-c7a8-4be3-a0d2-79539b2fe36d" path="/var/lib/kubelet/pods/33d78da4-c7a8-4be3-a0d2-79539b2fe36d/volumes" Nov 02 23:45:34.840118 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:34.840074 5114 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7699-22c384\" is forbidden: User \"system:node:ip-10-0-135-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-250.us-west-1.compute.internal' and this object" podUID="4811f132-cb2e-461b-8e9b-522eb00ecad9" pod="opendatahub/test-curl-7699-22c384" Nov 02 23:45:34.890804 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:34.890763 5114 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7699-751373\" is forbidden: User \"system:node:ip-10-0-135-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-250.us-west-1.compute.internal' and this object" podUID="33d78da4-c7a8-4be3-a0d2-79539b2fe36d" pod="opendatahub/test-curl-7699-751373" Nov 02 23:45:35.095954 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:35.095848 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-fef6cd" event={"ID":"a1061a95-f25f-4e3c-8e61-af12d8d717b8","Type":"ContainerStarted","Data":"034d9778ea08526b03c6a5d833a2f5cb82a5897041b3e710adc95837aefcab59"} Nov 02 23:45:35.095954 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:35.095904 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-fef6cd" event={"ID":"a1061a95-f25f-4e3c-8e61-af12d8d717b8","Type":"ContainerStarted","Data":"fd71762c3146f135441430015063f716419ad081dd804f1c227059914d180753"} Nov 02 23:45:35.207667 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:35.207604 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7699-fef6cd" podStartSLOduration=2.207585116 podStartE2EDuration="2.207585116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:35.207575613 +0000 UTC m=+3410.560608988" watchObservedRunningTime="2026-11-02 23:45:35.207585116 +0000 UTC m=+3410.560618491" Nov 02 23:45:36.333974 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:36.333943 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-22c384" Nov 02 23:45:36.443575 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:36.443443 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4811f132-cb2e-461b-8e9b-522eb00ecad9-kube-api-access-2cxc4\" (UniqueName: \"kubernetes.io/projected/4811f132-cb2e-461b-8e9b-522eb00ecad9-kube-api-access-2cxc4\") pod \"4811f132-cb2e-461b-8e9b-522eb00ecad9\" (UID: \"4811f132-cb2e-461b-8e9b-522eb00ecad9\") " Nov 02 23:45:36.446037 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:36.446012 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4811f132-cb2e-461b-8e9b-522eb00ecad9-kube-api-access-2cxc4" pod "4811f132-cb2e-461b-8e9b-522eb00ecad9" (UID: "4811f132-cb2e-461b-8e9b-522eb00ecad9"). InnerVolumeSpecName "kube-api-access-2cxc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:36.545286 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:36.545249 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cxc4\" (UniqueName: \"kubernetes.io/projected/4811f132-cb2e-461b-8e9b-522eb00ecad9-kube-api-access-2cxc4\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:36.600632 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:36.600592 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7699-fef6cd"] Nov 02 23:45:36.605609 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:36.605568 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7699-fef6cd"] Nov 02 23:45:36.651268 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:36.651227 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h"] Nov 02 23:45:36.741462 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:36.741371 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7699-305da1"] Nov 02 23:45:36.741899 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:36.741882 5114 cpu_manager.go:414] "containerMap: removing container" podUID="a1061a95-f25f-4e3c-8e61-af12d8d717b8" containerName="test-curl-7699-fef6cd" Nov 02 23:45:36.741946 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:36.741902 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1061a95-f25f-4e3c-8e61-af12d8d717b8" containerName="test-curl-7699-fef6cd" Nov 02 23:45:36.741995 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:36.741986 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1061a95-f25f-4e3c-8e61-af12d8d717b8" containerName="test-curl-7699-fef6cd" Nov 02 23:45:36.745130 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:36.745114 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-305da1" Nov 02 23:45:36.793830 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:36.793790 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4811f132-cb2e-461b-8e9b-522eb00ecad9" path="/var/lib/kubelet/pods/4811f132-cb2e-461b-8e9b-522eb00ecad9/volumes" Nov 02 23:45:36.801786 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:36.801750 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-305da1"] Nov 02 23:45:36.848380 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:36.848343 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mxth\" (UniqueName: \"kubernetes.io/projected/4f11ec7e-7a19-46d2-807d-2d0d51ef3f1d-kube-api-access-7mxth\") pod \"test-curl-7699-305da1\" (UID: \"4f11ec7e-7a19-46d2-807d-2d0d51ef3f1d\") " pod="opendatahub/test-curl-7699-305da1" Nov 02 23:45:36.949181 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:36.949143 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mxth\" (UniqueName: \"kubernetes.io/projected/4f11ec7e-7a19-46d2-807d-2d0d51ef3f1d-kube-api-access-7mxth\") pod \"test-curl-7699-305da1\" (UID: \"4f11ec7e-7a19-46d2-807d-2d0d51ef3f1d\") " pod="opendatahub/test-curl-7699-305da1" Nov 02 23:45:37.009599 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:37.009513 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mxth\" (UniqueName: \"kubernetes.io/projected/4f11ec7e-7a19-46d2-807d-2d0d51ef3f1d-kube-api-access-7mxth\") pod \"test-curl-7699-305da1\" (UID: \"4f11ec7e-7a19-46d2-807d-2d0d51ef3f1d\") " pod="opendatahub/test-curl-7699-305da1" Nov 02 23:45:37.058363 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:37.058319 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-305da1" Nov 02 23:45:37.112835 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:37.112794 5114 generic.go:356] "Generic (PLEG): container finished" podID="4811f132-cb2e-461b-8e9b-522eb00ecad9" containerID="9a1eb82c5cd549936a2fed2eb41b6fd40dab1c897d015a94a692f1b8ee95f43f" exitCode=137 Nov 02 23:45:37.113045 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:37.112920 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-22c384" Nov 02 23:45:37.113045 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:37.112945 5114 scope.go:122] "RemoveContainer" containerID="9a1eb82c5cd549936a2fed2eb41b6fd40dab1c897d015a94a692f1b8ee95f43f" Nov 02 23:45:37.113210 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:37.113119 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7699-fef6cd" podUID="a1061a95-f25f-4e3c-8e61-af12d8d717b8" containerName="test-curl-7699-fef6cd" containerID="cri-o://034d9778ea08526b03c6a5d833a2f5cb82a5897041b3e710adc95837aefcab59" gracePeriod=2 Nov 02 23:45:37.135546 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:37.128702 5114 scope.go:122] "RemoveContainer" containerID="9a1eb82c5cd549936a2fed2eb41b6fd40dab1c897d015a94a692f1b8ee95f43f" Nov 02 23:45:37.135546 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:37.129262 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a1eb82c5cd549936a2fed2eb41b6fd40dab1c897d015a94a692f1b8ee95f43f\": container with ID starting with 9a1eb82c5cd549936a2fed2eb41b6fd40dab1c897d015a94a692f1b8ee95f43f not found: ID does not exist" containerID="9a1eb82c5cd549936a2fed2eb41b6fd40dab1c897d015a94a692f1b8ee95f43f" Nov 02 23:45:37.135546 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:37.129293 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1eb82c5cd549936a2fed2eb41b6fd40dab1c897d015a94a692f1b8ee95f43f"} err="failed to get container status \"9a1eb82c5cd549936a2fed2eb41b6fd40dab1c897d015a94a692f1b8ee95f43f\": rpc error: code = NotFound desc = could not find container \"9a1eb82c5cd549936a2fed2eb41b6fd40dab1c897d015a94a692f1b8ee95f43f\": container with ID starting with 9a1eb82c5cd549936a2fed2eb41b6fd40dab1c897d015a94a692f1b8ee95f43f not found: ID does not exist" Nov 02 23:45:37.439795 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:37.439767 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-305da1"] Nov 02 23:45:37.442133 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:45:37.442105 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f11ec7e_7a19_46d2_807d_2d0d51ef3f1d.slice/crio-1287cabe5a28b1e6be71c205f6e7f015de108d24acc4babcfbac2c54f1a04908 WatchSource:0}: Error finding container 1287cabe5a28b1e6be71c205f6e7f015de108d24acc4babcfbac2c54f1a04908: Status 404 returned error can't find the container with id 1287cabe5a28b1e6be71c205f6e7f015de108d24acc4babcfbac2c54f1a04908 Nov 02 23:45:37.584814 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:37.584766 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:45:38.122780 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:38.122741 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-305da1" event={"ID":"4f11ec7e-7a19-46d2-807d-2d0d51ef3f1d","Type":"ContainerStarted","Data":"2a9ff80349b40d00af972281c514d9fbbbc0a1d18c50363205494231f51b829f"} Nov 02 23:45:38.122956 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:38.122791 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-305da1" event={"ID":"4f11ec7e-7a19-46d2-807d-2d0d51ef3f1d","Type":"ContainerStarted","Data":"1287cabe5a28b1e6be71c205f6e7f015de108d24acc4babcfbac2c54f1a04908"} Nov 02 23:45:38.235735 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:38.235671 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7699-305da1" podStartSLOduration=2.235655835 podStartE2EDuration="2.235655835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:38.234404281 +0000 UTC m=+3413.587437656" watchObservedRunningTime="2026-11-02 23:45:38.235655835 +0000 UTC m=+3413.588689210" Nov 02 23:45:39.381558 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:39.381471 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-fef6cd" Nov 02 23:45:39.477088 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:39.477049 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1061a95-f25f-4e3c-8e61-af12d8d717b8-kube-api-access-pq6m8\" (UniqueName: \"kubernetes.io/projected/a1061a95-f25f-4e3c-8e61-af12d8d717b8-kube-api-access-pq6m8\") pod \"a1061a95-f25f-4e3c-8e61-af12d8d717b8\" (UID: \"a1061a95-f25f-4e3c-8e61-af12d8d717b8\") " Nov 02 23:45:39.479437 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:39.479411 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1061a95-f25f-4e3c-8e61-af12d8d717b8-kube-api-access-pq6m8" pod "a1061a95-f25f-4e3c-8e61-af12d8d717b8" (UID: "a1061a95-f25f-4e3c-8e61-af12d8d717b8"). InnerVolumeSpecName "kube-api-access-pq6m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:39.579011 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:39.578965 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq6m8\" (UniqueName: \"kubernetes.io/projected/a1061a95-f25f-4e3c-8e61-af12d8d717b8-kube-api-access-pq6m8\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:39.710912 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:39.710818 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7699-305da1"] Nov 02 23:45:39.717505 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:39.717443 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7699-305da1"] Nov 02 23:45:40.135263 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:40.135229 5114 generic.go:356] "Generic (PLEG): container finished" podID="a1061a95-f25f-4e3c-8e61-af12d8d717b8" containerID="034d9778ea08526b03c6a5d833a2f5cb82a5897041b3e710adc95837aefcab59" exitCode=137 Nov 02 23:45:40.135443 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:40.135287 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-fef6cd" Nov 02 23:45:40.135443 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:40.135324 5114 scope.go:122] "RemoveContainer" containerID="034d9778ea08526b03c6a5d833a2f5cb82a5897041b3e710adc95837aefcab59" Nov 02 23:45:40.135613 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:40.135587 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7699-305da1" podUID="4f11ec7e-7a19-46d2-807d-2d0d51ef3f1d" containerName="test-curl-7699-305da1" containerID="cri-o://2a9ff80349b40d00af972281c514d9fbbbc0a1d18c50363205494231f51b829f" gracePeriod=2 Nov 02 23:45:40.146389 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:40.146367 5114 scope.go:122] "RemoveContainer" containerID="034d9778ea08526b03c6a5d833a2f5cb82a5897041b3e710adc95837aefcab59" Nov 02 23:45:40.146792 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:40.146767 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"034d9778ea08526b03c6a5d833a2f5cb82a5897041b3e710adc95837aefcab59\": container with ID starting with 034d9778ea08526b03c6a5d833a2f5cb82a5897041b3e710adc95837aefcab59 not found: ID does not exist" containerID="034d9778ea08526b03c6a5d833a2f5cb82a5897041b3e710adc95837aefcab59" Nov 02 23:45:40.146849 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:40.146803 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"034d9778ea08526b03c6a5d833a2f5cb82a5897041b3e710adc95837aefcab59"} err="failed to get container status \"034d9778ea08526b03c6a5d833a2f5cb82a5897041b3e710adc95837aefcab59\": rpc error: code = NotFound desc = could not find container \"034d9778ea08526b03c6a5d833a2f5cb82a5897041b3e710adc95837aefcab59\": container with ID starting with 034d9778ea08526b03c6a5d833a2f5cb82a5897041b3e710adc95837aefcab59 not found: ID does not exist" Nov 02 23:45:40.176053 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:40.176015 5114 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1061a95_f25f_4e3c_8e61_af12d8d717b8.slice\": RecentStats: unable to find data in memory cache]" Nov 02 23:45:40.186327 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:40.186291 5114 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7699-305da1\" is forbidden: User \"system:node:ip-10-0-135-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-250.us-west-1.compute.internal' and this object" podUID="4f11ec7e-7a19-46d2-807d-2d0d51ef3f1d" pod="opendatahub/test-curl-7699-305da1" Nov 02 23:45:40.236992 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:40.236939 5114 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7699-305da1\" is forbidden: User \"system:node:ip-10-0-135-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-250.us-west-1.compute.internal' and this object" podUID="4f11ec7e-7a19-46d2-807d-2d0d51ef3f1d" pod="opendatahub/test-curl-7699-305da1" Nov 02 23:45:40.287648 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:40.287605 5114 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7699-fef6cd\" is forbidden: User \"system:node:ip-10-0-135-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-250.us-west-1.compute.internal' and this object" podUID="a1061a95-f25f-4e3c-8e61-af12d8d717b8" pod="opendatahub/test-curl-7699-fef6cd" Nov 02 23:45:40.792609 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:40.792568 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1061a95-f25f-4e3c-8e61-af12d8d717b8" path="/var/lib/kubelet/pods/a1061a95-f25f-4e3c-8e61-af12d8d717b8/volumes" Nov 02 23:45:42.386496 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:42.386409 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-305da1" Nov 02 23:45:42.514931 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:42.514824 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f11ec7e-7a19-46d2-807d-2d0d51ef3f1d-kube-api-access-7mxth\" (UniqueName: \"kubernetes.io/projected/4f11ec7e-7a19-46d2-807d-2d0d51ef3f1d-kube-api-access-7mxth\") pod \"4f11ec7e-7a19-46d2-807d-2d0d51ef3f1d\" (UID: \"4f11ec7e-7a19-46d2-807d-2d0d51ef3f1d\") " Nov 02 23:45:42.517353 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:42.517302 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f11ec7e-7a19-46d2-807d-2d0d51ef3f1d-kube-api-access-7mxth" pod "4f11ec7e-7a19-46d2-807d-2d0d51ef3f1d" (UID: "4f11ec7e-7a19-46d2-807d-2d0d51ef3f1d"). InnerVolumeSpecName "kube-api-access-7mxth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:42.617029 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:42.616989 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mxth\" (UniqueName: \"kubernetes.io/projected/4f11ec7e-7a19-46d2-807d-2d0d51ef3f1d-kube-api-access-7mxth\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:42.793635 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:42.793597 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f11ec7e-7a19-46d2-807d-2d0d51ef3f1d" path="/var/lib/kubelet/pods/4f11ec7e-7a19-46d2-807d-2d0d51ef3f1d/volumes" Nov 02 23:45:43.157996 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:43.157887 5114 generic.go:356] "Generic (PLEG): container finished" podID="4f11ec7e-7a19-46d2-807d-2d0d51ef3f1d" containerID="2a9ff80349b40d00af972281c514d9fbbbc0a1d18c50363205494231f51b829f" exitCode=137 Nov 02 23:45:43.157996 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:43.157940 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-305da1" Nov 02 23:45:43.157996 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:43.157965 5114 scope.go:122] "RemoveContainer" containerID="2a9ff80349b40d00af972281c514d9fbbbc0a1d18c50363205494231f51b829f" Nov 02 23:45:43.169039 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:43.169013 5114 scope.go:122] "RemoveContainer" containerID="2a9ff80349b40d00af972281c514d9fbbbc0a1d18c50363205494231f51b829f" Nov 02 23:45:43.169442 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:43.169418 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a9ff80349b40d00af972281c514d9fbbbc0a1d18c50363205494231f51b829f\": container with ID starting with 2a9ff80349b40d00af972281c514d9fbbbc0a1d18c50363205494231f51b829f not found: ID does not exist" containerID="2a9ff80349b40d00af972281c514d9fbbbc0a1d18c50363205494231f51b829f" Nov 02 23:45:43.169527 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:43.169454 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a9ff80349b40d00af972281c514d9fbbbc0a1d18c50363205494231f51b829f"} err="failed to get container status \"2a9ff80349b40d00af972281c514d9fbbbc0a1d18c50363205494231f51b829f\": rpc error: code = NotFound desc = could not find container \"2a9ff80349b40d00af972281c514d9fbbbc0a1d18c50363205494231f51b829f\": container with ID starting with 2a9ff80349b40d00af972281c514d9fbbbc0a1d18c50363205494231f51b829f not found: ID does not exist" Nov 02 23:45:45.681859 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:45.681823 5114 scope.go:122] "RemoveContainer" containerID="f7319cffef8ab05c787c4af6a33e24af546d50f89b75ecae45affef9985e0c2c" Nov 02 23:45:48.312780 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:48.312745 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f"] Nov 02 23:45:51.673264 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:51.673193 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" podUID="356fc994-5116-4aa2-9d6f-159c641f6fcd" containerName="main" containerID="cri-o://216fa9ca31ab0517bcd5910e6238cf25898a8b60fffd1e2f5b7ee7d1efdabdc7" gracePeriod=45 Nov 02 23:45:51.937215 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:51.937138 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:45:52.040929 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.040885 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-tmp-dir\") pod \"356fc994-5116-4aa2-9d6f-159c641f6fcd\" (UID: \"356fc994-5116-4aa2-9d6f-159c641f6fcd\") " Nov 02 23:45:52.041123 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.040993 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/356fc994-5116-4aa2-9d6f-159c641f6fcd-kube-api-access-ctz9v\" (UniqueName: \"kubernetes.io/projected/356fc994-5116-4aa2-9d6f-159c641f6fcd-kube-api-access-ctz9v\") pod \"356fc994-5116-4aa2-9d6f-159c641f6fcd\" (UID: \"356fc994-5116-4aa2-9d6f-159c641f6fcd\") " Nov 02 23:45:52.041123 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.041025 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-home\" (UniqueName: \"kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-home\") pod \"356fc994-5116-4aa2-9d6f-159c641f6fcd\" (UID: \"356fc994-5116-4aa2-9d6f-159c641f6fcd\") " Nov 02 23:45:52.041123 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.041066 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-dshm\") pod \"356fc994-5116-4aa2-9d6f-159c641f6fcd\" (UID: \"356fc994-5116-4aa2-9d6f-159c641f6fcd\") " Nov 02 23:45:52.041299 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.041244 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-tmp-dir" pod "356fc994-5116-4aa2-9d6f-159c641f6fcd" (UID: "356fc994-5116-4aa2-9d6f-159c641f6fcd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:52.041299 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.041260 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/356fc994-5116-4aa2-9d6f-159c641f6fcd-tls-certs\" (UniqueName: \"kubernetes.io/secret/356fc994-5116-4aa2-9d6f-159c641f6fcd-tls-certs\") pod \"356fc994-5116-4aa2-9d6f-159c641f6fcd\" (UID: \"356fc994-5116-4aa2-9d6f-159c641f6fcd\") " Nov 02 23:45:52.041299 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.041279 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-home" pod "356fc994-5116-4aa2-9d6f-159c641f6fcd" (UID: "356fc994-5116-4aa2-9d6f-159c641f6fcd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:52.041457 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.041343 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-model-cache\") pod \"356fc994-5116-4aa2-9d6f-159c641f6fcd\" (UID: \"356fc994-5116-4aa2-9d6f-159c641f6fcd\") " Nov 02 23:45:52.041666 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.041638 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-model-cache" pod "356fc994-5116-4aa2-9d6f-159c641f6fcd" (UID: "356fc994-5116-4aa2-9d6f-159c641f6fcd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:52.041968 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.041877 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:52.042074 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.041970 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:52.042074 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.041986 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:52.043558 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.043536 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-dshm" pod "356fc994-5116-4aa2-9d6f-159c641f6fcd" (UID: "356fc994-5116-4aa2-9d6f-159c641f6fcd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:52.043630 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.043572 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/356fc994-5116-4aa2-9d6f-159c641f6fcd-kube-api-access-ctz9v" pod "356fc994-5116-4aa2-9d6f-159c641f6fcd" (UID: "356fc994-5116-4aa2-9d6f-159c641f6fcd"). InnerVolumeSpecName "kube-api-access-ctz9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:52.043676 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.043641 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/356fc994-5116-4aa2-9d6f-159c641f6fcd-tls-certs" pod "356fc994-5116-4aa2-9d6f-159c641f6fcd" (UID: "356fc994-5116-4aa2-9d6f-159c641f6fcd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:45:52.143045 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.143001 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/356fc994-5116-4aa2-9d6f-159c641f6fcd-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:52.143045 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.143031 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/356fc994-5116-4aa2-9d6f-159c641f6fcd-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:52.143045 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.143042 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctz9v\" (UniqueName: \"kubernetes.io/projected/356fc994-5116-4aa2-9d6f-159c641f6fcd-kube-api-access-ctz9v\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:52.218216 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.218123 5114 generic.go:356] "Generic (PLEG): container finished" podID="356fc994-5116-4aa2-9d6f-159c641f6fcd" containerID="216fa9ca31ab0517bcd5910e6238cf25898a8b60fffd1e2f5b7ee7d1efdabdc7" exitCode=0 Nov 02 23:45:52.218216 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.218166 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" Nov 02 23:45:52.218216 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.218188 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" event={"ID":"356fc994-5116-4aa2-9d6f-159c641f6fcd","Type":"ContainerDied","Data":"216fa9ca31ab0517bcd5910e6238cf25898a8b60fffd1e2f5b7ee7d1efdabdc7"} Nov 02 23:45:52.218466 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.218234 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h" event={"ID":"356fc994-5116-4aa2-9d6f-159c641f6fcd","Type":"ContainerDied","Data":"2668d7bc1dab6e5089b4756fae4d1b2435a772d16b7e05998d8e4cc379f2ce73"} Nov 02 23:45:52.218466 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.218258 5114 scope.go:122] "RemoveContainer" containerID="216fa9ca31ab0517bcd5910e6238cf25898a8b60fffd1e2f5b7ee7d1efdabdc7" Nov 02 23:45:52.228728 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.228706 5114 scope.go:122] "RemoveContainer" containerID="216fa9ca31ab0517bcd5910e6238cf25898a8b60fffd1e2f5b7ee7d1efdabdc7" Nov 02 23:45:52.229064 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:45:52.229044 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"216fa9ca31ab0517bcd5910e6238cf25898a8b60fffd1e2f5b7ee7d1efdabdc7\": container with ID starting with 216fa9ca31ab0517bcd5910e6238cf25898a8b60fffd1e2f5b7ee7d1efdabdc7 not found: ID does not exist" containerID="216fa9ca31ab0517bcd5910e6238cf25898a8b60fffd1e2f5b7ee7d1efdabdc7" Nov 02 23:45:52.229135 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.229073 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"216fa9ca31ab0517bcd5910e6238cf25898a8b60fffd1e2f5b7ee7d1efdabdc7"} err="failed to get container status \"216fa9ca31ab0517bcd5910e6238cf25898a8b60fffd1e2f5b7ee7d1efdabdc7\": rpc error: code = NotFound desc = could not find container \"216fa9ca31ab0517bcd5910e6238cf25898a8b60fffd1e2f5b7ee7d1efdabdc7\": container with ID starting with 216fa9ca31ab0517bcd5910e6238cf25898a8b60fffd1e2f5b7ee7d1efdabdc7 not found: ID does not exist" Nov 02 23:45:52.381945 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.381904 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h"] Nov 02 23:45:52.386044 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.386014 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-875049/autores-model-875049-kserve-5f4c5f78c7-rgx5h"] Nov 02 23:45:52.793732 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:52.793694 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="356fc994-5116-4aa2-9d6f-159c641f6fcd" path="/var/lib/kubelet/pods/356fc994-5116-4aa2-9d6f-159c641f6fcd/volumes" Nov 02 23:45:54.038388 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.038355 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj"] Nov 02 23:45:54.038852 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.038835 5114 cpu_manager.go:414] "containerMap: removing container" podUID="356fc994-5116-4aa2-9d6f-159c641f6fcd" containerName="main" Nov 02 23:45:54.038898 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.038854 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="356fc994-5116-4aa2-9d6f-159c641f6fcd" containerName="main" Nov 02 23:45:54.038898 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.038881 5114 cpu_manager.go:414] "containerMap: removing container" podUID="4f11ec7e-7a19-46d2-807d-2d0d51ef3f1d" containerName="test-curl-7699-305da1" Nov 02 23:45:54.038898 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.038886 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f11ec7e-7a19-46d2-807d-2d0d51ef3f1d" containerName="test-curl-7699-305da1" Nov 02 23:45:54.039000 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.038971 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="356fc994-5116-4aa2-9d6f-159c641f6fcd" containerName="main" Nov 02 23:45:54.039000 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.038982 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f11ec7e-7a19-46d2-807d-2d0d51ef3f1d" containerName="test-curl-7699-305da1" Nov 02 23:45:54.043414 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.043395 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj" Nov 02 23:45:54.064694 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.064647 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhvz2\" (UniqueName: \"kubernetes.io/projected/7773e0dc-6fee-4e97-9db3-9106268ac5c4-kube-api-access-nhvz2\") pod \"payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj\" (UID: \"7773e0dc-6fee-4e97-9db3-9106268ac5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj" Nov 02 23:45:54.064694 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.064700 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7773e0dc-6fee-4e97-9db3-9106268ac5c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj\" (UID: \"7773e0dc-6fee-4e97-9db3-9106268ac5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj" Nov 02 23:45:54.098457 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.098420 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-3058ee\"" Nov 02 23:45:54.099241 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.099221 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-3058ee-dockercfg-9qdr4\"" Nov 02 23:45:54.152861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.152812 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj"] Nov 02 23:45:54.165900 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.165861 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhvz2\" (UniqueName: \"kubernetes.io/projected/7773e0dc-6fee-4e97-9db3-9106268ac5c4-kube-api-access-nhvz2\") pod \"payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj\" (UID: \"7773e0dc-6fee-4e97-9db3-9106268ac5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj" Nov 02 23:45:54.166092 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.165917 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7773e0dc-6fee-4e97-9db3-9106268ac5c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj\" (UID: \"7773e0dc-6fee-4e97-9db3-9106268ac5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj" Nov 02 23:45:54.166733 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.166712 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7773e0dc-6fee-4e97-9db3-9106268ac5c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj\" (UID: \"7773e0dc-6fee-4e97-9db3-9106268ac5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj" Nov 02 23:45:54.191202 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.191157 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz"] Nov 02 23:45:54.194864 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.194845 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz" Nov 02 23:45:54.226692 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.226659 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhvz2\" (UniqueName: \"kubernetes.io/projected/7773e0dc-6fee-4e97-9db3-9106268ac5c4-kube-api-access-nhvz2\") pod \"payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj\" (UID: \"7773e0dc-6fee-4e97-9db3-9106268ac5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj" Nov 02 23:45:54.252880 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.252841 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz"] Nov 02 23:45:54.267580 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.267473 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/084ed54a-9cca-405e-a481-6933da8821dc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz\" (UID: \"084ed54a-9cca-405e-a481-6933da8821dc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz" Nov 02 23:45:54.267770 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.267597 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c84mw\" (UniqueName: \"kubernetes.io/projected/084ed54a-9cca-405e-a481-6933da8821dc-kube-api-access-c84mw\") pod \"payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz\" (UID: \"084ed54a-9cca-405e-a481-6933da8821dc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz" Nov 02 23:45:54.358369 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.358228 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj" Nov 02 23:45:54.368905 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.368876 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/084ed54a-9cca-405e-a481-6933da8821dc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz\" (UID: \"084ed54a-9cca-405e-a481-6933da8821dc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz" Nov 02 23:45:54.369098 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.368934 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c84mw\" (UniqueName: \"kubernetes.io/projected/084ed54a-9cca-405e-a481-6933da8821dc-kube-api-access-c84mw\") pod \"payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz\" (UID: \"084ed54a-9cca-405e-a481-6933da8821dc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz" Nov 02 23:45:54.369670 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.369651 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/084ed54a-9cca-405e-a481-6933da8821dc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz\" (UID: \"084ed54a-9cca-405e-a481-6933da8821dc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz" Nov 02 23:45:54.436465 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.436428 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c84mw\" (UniqueName: \"kubernetes.io/projected/084ed54a-9cca-405e-a481-6933da8821dc-kube-api-access-c84mw\") pod \"payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz\" (UID: \"084ed54a-9cca-405e-a481-6933da8821dc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz" Nov 02 23:45:54.509262 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.509214 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz" Nov 02 23:45:54.735038 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.735008 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj"] Nov 02 23:45:54.737390 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:45:54.737359 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7773e0dc_6fee_4e97_9db3_9106268ac5c4.slice/crio-431df346754247a47622d33d418a7c98ace2df5002053a4529398eaf15a0c4ee WatchSource:0}: Error finding container 431df346754247a47622d33d418a7c98ace2df5002053a4529398eaf15a0c4ee: Status 404 returned error can't find the container with id 431df346754247a47622d33d418a7c98ace2df5002053a4529398eaf15a0c4ee Nov 02 23:45:54.890624 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:54.890596 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz"] Nov 02 23:45:54.892847 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:45:54.892814 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084ed54a_9cca_405e_a481_6933da8821dc.slice/crio-7af11e96eb74fecfb9f3d5027d98fbbb78e1e5edcfb46b4605c2ed5b791e18d6 WatchSource:0}: Error finding container 7af11e96eb74fecfb9f3d5027d98fbbb78e1e5edcfb46b4605c2ed5b791e18d6: Status 404 returned error can't find the container with id 7af11e96eb74fecfb9f3d5027d98fbbb78e1e5edcfb46b4605c2ed5b791e18d6 Nov 02 23:45:55.237690 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:55.237645 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz" event={"ID":"084ed54a-9cca-405e-a481-6933da8821dc","Type":"ContainerStarted","Data":"f44671821d5cddfe15685c9a8ab0016ca910360025b58ace5ad6a1ec0992ed68"} Nov 02 23:45:55.238201 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:55.237728 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz" Nov 02 23:45:55.238201 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:55.237756 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz" event={"ID":"084ed54a-9cca-405e-a481-6933da8821dc","Type":"ContainerStarted","Data":"7af11e96eb74fecfb9f3d5027d98fbbb78e1e5edcfb46b4605c2ed5b791e18d6"} Nov 02 23:45:55.239687 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:55.239657 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj" event={"ID":"7773e0dc-6fee-4e97-9db3-9106268ac5c4","Type":"ContainerStarted","Data":"c151504f67df962367f2b889c810765d6a1b2be6d9ef7732e7611d6fcb70cd41"} Nov 02 23:45:55.239848 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:55.239703 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj" event={"ID":"7773e0dc-6fee-4e97-9db3-9106268ac5c4","Type":"ContainerStarted","Data":"431df346754247a47622d33d418a7c98ace2df5002053a4529398eaf15a0c4ee"} Nov 02 23:45:55.239848 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:55.239738 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj" Nov 02 23:45:55.353502 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:55.353404 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz" podStartSLOduration=1.353381538 podStartE2EDuration="1.353381538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:55.352111701 +0000 UTC m=+3430.705145077" watchObservedRunningTime="2026-11-02 23:45:55.353381538 +0000 UTC m=+3430.706414918" Nov 02 23:45:55.467357 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:55.467267 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj" podStartSLOduration=2.467243921 podStartE2EDuration="2.467243921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:55.466562116 +0000 UTC m=+3430.819595492" watchObservedRunningTime="2026-11-02 23:45:55.467243921 +0000 UTC m=+3430.820277296" Nov 02 23:45:55.764798 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:45:55.764754 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn"] Nov 02 23:46:03.334223 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:03.334156 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" podUID="c922368c-26c9-4ed6-9f52-3b7df5f8ce73" containerName="main" containerID="cri-o://4c83dc4474d2d38bc93f4507ca39dc64311d31a281be3eeb7b311f1c0c6c36dd" gracePeriod=45 Nov 02 23:46:03.602352 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:03.602323 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:46:03.679122 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:03.679082 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-home\" (UniqueName: \"kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-home\") pod \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\" (UID: \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\") " Nov 02 23:46:03.679332 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:03.679137 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-model-cache\") pod \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\" (UID: \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\") " Nov 02 23:46:03.679332 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:03.679197 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-tls-certs\" (UniqueName: \"kubernetes.io/secret/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-tls-certs\") pod \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\" (UID: \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\") " Nov 02 23:46:03.679332 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:03.679217 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-dshm\") pod \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\" (UID: \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\") " Nov 02 23:46:03.679332 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:03.679263 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-kube-api-access-w4l9b\" (UniqueName: \"kubernetes.io/projected/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-kube-api-access-w4l9b\") pod \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\" (UID: \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\") " Nov 02 23:46:03.679607 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:03.679389 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-model-cache" pod "c922368c-26c9-4ed6-9f52-3b7df5f8ce73" (UID: "c922368c-26c9-4ed6-9f52-3b7df5f8ce73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:03.679607 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:03.679404 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-home" pod "c922368c-26c9-4ed6-9f52-3b7df5f8ce73" (UID: "c922368c-26c9-4ed6-9f52-3b7df5f8ce73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:03.679607 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:03.679416 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-tmp-dir\") pod \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\" (UID: \"c922368c-26c9-4ed6-9f52-3b7df5f8ce73\") " Nov 02 23:46:03.679766 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:03.679737 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-tmp-dir" pod "c922368c-26c9-4ed6-9f52-3b7df5f8ce73" (UID: "c922368c-26c9-4ed6-9f52-3b7df5f8ce73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:03.679885 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:03.679869 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:03.679966 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:03.679887 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:03.679966 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:03.679897 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:03.681877 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:03.681847 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-tls-certs" pod "c922368c-26c9-4ed6-9f52-3b7df5f8ce73" (UID: "c922368c-26c9-4ed6-9f52-3b7df5f8ce73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:46:03.681877 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:03.681850 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-dshm" pod "c922368c-26c9-4ed6-9f52-3b7df5f8ce73" (UID: "c922368c-26c9-4ed6-9f52-3b7df5f8ce73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:03.682078 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:03.681925 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-kube-api-access-w4l9b" pod "c922368c-26c9-4ed6-9f52-3b7df5f8ce73" (UID: "c922368c-26c9-4ed6-9f52-3b7df5f8ce73"). InnerVolumeSpecName "kube-api-access-w4l9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:46:03.780551 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:03.780475 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:03.780551 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:03.780543 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:03.780551 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:03.780555 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4l9b\" (UniqueName: \"kubernetes.io/projected/c922368c-26c9-4ed6-9f52-3b7df5f8ce73-kube-api-access-w4l9b\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:04.291955 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:04.291919 5114 generic.go:356] "Generic (PLEG): container finished" podID="c922368c-26c9-4ed6-9f52-3b7df5f8ce73" containerID="4c83dc4474d2d38bc93f4507ca39dc64311d31a281be3eeb7b311f1c0c6c36dd" exitCode=0 Nov 02 23:46:04.292189 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:04.291970 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" Nov 02 23:46:04.292189 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:04.292008 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" event={"ID":"c922368c-26c9-4ed6-9f52-3b7df5f8ce73","Type":"ContainerDied","Data":"4c83dc4474d2d38bc93f4507ca39dc64311d31a281be3eeb7b311f1c0c6c36dd"} Nov 02 23:46:04.292189 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:04.292061 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f" event={"ID":"c922368c-26c9-4ed6-9f52-3b7df5f8ce73","Type":"ContainerDied","Data":"c309d29281dfe9e027452205f8ff45899514b883f90729326fda1e0e863740ec"} Nov 02 23:46:04.292189 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:04.292082 5114 scope.go:122] "RemoveContainer" containerID="4c83dc4474d2d38bc93f4507ca39dc64311d31a281be3eeb7b311f1c0c6c36dd" Nov 02 23:46:04.302275 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:04.302252 5114 scope.go:122] "RemoveContainer" containerID="4c83dc4474d2d38bc93f4507ca39dc64311d31a281be3eeb7b311f1c0c6c36dd" Nov 02 23:46:04.302696 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:46:04.302665 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c83dc4474d2d38bc93f4507ca39dc64311d31a281be3eeb7b311f1c0c6c36dd\": container with ID starting with 4c83dc4474d2d38bc93f4507ca39dc64311d31a281be3eeb7b311f1c0c6c36dd not found: ID does not exist" containerID="4c83dc4474d2d38bc93f4507ca39dc64311d31a281be3eeb7b311f1c0c6c36dd" Nov 02 23:46:04.302838 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:04.302706 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c83dc4474d2d38bc93f4507ca39dc64311d31a281be3eeb7b311f1c0c6c36dd"} err="failed to get container status \"4c83dc4474d2d38bc93f4507ca39dc64311d31a281be3eeb7b311f1c0c6c36dd\": rpc error: code = NotFound desc = could not find container \"4c83dc4474d2d38bc93f4507ca39dc64311d31a281be3eeb7b311f1c0c6c36dd\": container with ID starting with 4c83dc4474d2d38bc93f4507ca39dc64311d31a281be3eeb7b311f1c0c6c36dd not found: ID does not exist" Nov 02 23:46:04.455519 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:04.455467 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f"] Nov 02 23:46:04.505178 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:04.505138 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8d3cb/auth-test-model-b8d3cb-kserve-6c7755c5f7-bvn8f"] Nov 02 23:46:04.796341 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:04.796303 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c922368c-26c9-4ed6-9f52-3b7df5f8ce73" path="/var/lib/kubelet/pods/c922368c-26c9-4ed6-9f52-3b7df5f8ce73/volumes" Nov 02 23:46:06.249038 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.249007 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj" Nov 02 23:46:06.249525 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.249078 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz" Nov 02 23:46:06.452516 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.452462 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j"] Nov 02 23:46:06.452998 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.452978 5114 cpu_manager.go:414] "containerMap: removing container" podUID="c922368c-26c9-4ed6-9f52-3b7df5f8ce73" containerName="main" Nov 02 23:46:06.453122 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.452998 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c922368c-26c9-4ed6-9f52-3b7df5f8ce73" containerName="main" Nov 02 23:46:06.453182 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.453141 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="c922368c-26c9-4ed6-9f52-3b7df5f8ce73" containerName="main" Nov 02 23:46:06.458612 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.458585 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.512569 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.511131 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0400368-a485-486f-b433-3c456aaddff9-istiod-ca-cert\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.512569 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.511190 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-workload-socket\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.512569 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.511220 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-credential-socket\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.512569 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.511281 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-istio-data\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.512569 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.511341 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-workload-certs\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.512569 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.511375 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfb54\" (UniqueName: \"kubernetes.io/projected/d0400368-a485-486f-b433-3c456aaddff9-kube-api-access-gfb54\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.512569 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.511450 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0400368-a485-486f-b433-3c456aaddff9-istio-token\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.512569 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.511533 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0400368-a485-486f-b433-3c456aaddff9-istio-podinfo\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.512569 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.511564 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-istio-envoy\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.520532 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.513871 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-1498df-openshift-default-dockercfg-qrxbq\"" Nov 02 23:46:06.520532 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.518010 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j"] Nov 02 23:46:06.612890 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.612845 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0400368-a485-486f-b433-3c456aaddff9-istio-podinfo\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.613115 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.612906 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-istio-envoy\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.613115 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.612972 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0400368-a485-486f-b433-3c456aaddff9-istiod-ca-cert\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.613115 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.612997 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-workload-socket\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.613115 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.613020 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-credential-socket\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.613115 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.613088 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-istio-data\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.613385 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.613151 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-workload-certs\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.613385 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.613194 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfb54\" (UniqueName: \"kubernetes.io/projected/d0400368-a485-486f-b433-3c456aaddff9-kube-api-access-gfb54\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.613385 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.613287 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0400368-a485-486f-b433-3c456aaddff9-istio-token\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.614619 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.614323 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-credential-socket\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.614619 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.614567 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-istio-data\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.614829 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.614654 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-workload-socket\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.614954 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.614933 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0400368-a485-486f-b433-3c456aaddff9-istiod-ca-cert\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.615265 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.615246 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-workload-certs\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.616534 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.616503 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0400368-a485-486f-b433-3c456aaddff9-istio-podinfo\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.619504 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.619362 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-istio-envoy\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.674170 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.674128 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0400368-a485-486f-b433-3c456aaddff9-istio-token\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.674344 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.674182 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfb54\" (UniqueName: \"kubernetes.io/projected/d0400368-a485-486f-b433-3c456aaddff9-kube-api-access-gfb54\") pod \"e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:06.771967 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:06.771863 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:07.211029 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:07.210984 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j"] Nov 02 23:46:07.214941 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:46:07.214910 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0400368_a485_486f_b433_3c456aaddff9.slice/crio-56a0379782b03853f321175f7bbd43daca543efe40c72c5cec68690223713878 WatchSource:0}: Error finding container 56a0379782b03853f321175f7bbd43daca543efe40c72c5cec68690223713878: Status 404 returned error can't find the container with id 56a0379782b03853f321175f7bbd43daca543efe40c72c5cec68690223713878 Nov 02 23:46:07.217870 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:07.217832 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:46:07.218002 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:07.217905 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:46:07.218002 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:07.217936 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:46:07.314915 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:07.314860 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" event={"ID":"d0400368-a485-486f-b433-3c456aaddff9","Type":"ContainerStarted","Data":"7bb850e4a88cf684fae83c5159cbecd1e9b3c0c3a95d5407a969ea0156cc5623"} Nov 02 23:46:07.315364 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:07.314932 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" event={"ID":"d0400368-a485-486f-b433-3c456aaddff9","Type":"ContainerStarted","Data":"56a0379782b03853f321175f7bbd43daca543efe40c72c5cec68690223713878"} Nov 02 23:46:07.437714 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:07.437651 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" podStartSLOduration=1.437634734 podStartE2EDuration="1.437634734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:46:07.435474736 +0000 UTC m=+3442.788508112" watchObservedRunningTime="2026-11-02 23:46:07.437634734 +0000 UTC m=+3442.790668109" Nov 02 23:46:07.589668 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:46:07.589623 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:46:07.772013 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:07.771953 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:08.778981 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:08.778943 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:09.045656 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:09.045579 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c516707-ds7zn"] Nov 02 23:46:09.048203 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:09.048172 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c516707-ds7zn"] Nov 02 23:46:09.326938 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:09.326838 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:09.328094 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:09.328072 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:46:10.786727 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:10.786680 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" podUID="195e0f31-6e9c-4861-aa00-2dcb8e5c17de" containerName="main" containerID="cri-o://d0cfedab1e5cf11b0ad5855650a46bee2afbccd9e3adecb378e1a28c4399599f" gracePeriod=45 Nov 02 23:46:10.794925 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:10.794894 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02495134-1dc0-44f6-a856-5c8f8d2ec72c" path="/var/lib/kubelet/pods/02495134-1dc0-44f6-a856-5c8f8d2ec72c/volumes" Nov 02 23:46:11.063221 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.063188 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:46:11.164809 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.164774 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-model-cache\") pod \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\" (UID: \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\") " Nov 02 23:46:11.165026 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.164845 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-tls-certs\" (UniqueName: \"kubernetes.io/secret/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-tls-certs\") pod \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\" (UID: \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\") " Nov 02 23:46:11.165026 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.164873 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-dshm\") pod \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\" (UID: \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\") " Nov 02 23:46:11.165026 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.164988 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-kube-api-access-rh4xq\" (UniqueName: \"kubernetes.io/projected/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-kube-api-access-rh4xq\") pod \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\" (UID: \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\") " Nov 02 23:46:11.165269 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.165092 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-tmp-dir\") pod \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\" (UID: \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\") " Nov 02 23:46:11.165269 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.165117 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-model-cache" pod "195e0f31-6e9c-4861-aa00-2dcb8e5c17de" (UID: "195e0f31-6e9c-4861-aa00-2dcb8e5c17de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:11.165269 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.165233 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-home\" (UniqueName: \"kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-home\") pod \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\" (UID: \"195e0f31-6e9c-4861-aa00-2dcb8e5c17de\") " Nov 02 23:46:11.165419 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.165319 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-tmp-dir" pod "195e0f31-6e9c-4861-aa00-2dcb8e5c17de" (UID: "195e0f31-6e9c-4861-aa00-2dcb8e5c17de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:11.165516 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.165459 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-home" pod "195e0f31-6e9c-4861-aa00-2dcb8e5c17de" (UID: "195e0f31-6e9c-4861-aa00-2dcb8e5c17de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:11.165693 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.165674 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:11.165756 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.165699 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:11.165756 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.165713 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:11.167451 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.167430 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-kube-api-access-rh4xq" pod "195e0f31-6e9c-4861-aa00-2dcb8e5c17de" (UID: "195e0f31-6e9c-4861-aa00-2dcb8e5c17de"). InnerVolumeSpecName "kube-api-access-rh4xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:46:11.167749 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.167723 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-dshm" pod "195e0f31-6e9c-4861-aa00-2dcb8e5c17de" (UID: "195e0f31-6e9c-4861-aa00-2dcb8e5c17de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:11.167749 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.167736 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-tls-certs" pod "195e0f31-6e9c-4861-aa00-2dcb8e5c17de" (UID: "195e0f31-6e9c-4861-aa00-2dcb8e5c17de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:46:11.267158 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.267126 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:11.267158 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.267155 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:11.267158 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.267166 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh4xq\" (UniqueName: \"kubernetes.io/projected/195e0f31-6e9c-4861-aa00-2dcb8e5c17de-kube-api-access-rh4xq\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:11.338382 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.338290 5114 generic.go:356] "Generic (PLEG): container finished" podID="195e0f31-6e9c-4861-aa00-2dcb8e5c17de" containerID="d0cfedab1e5cf11b0ad5855650a46bee2afbccd9e3adecb378e1a28c4399599f" exitCode=0 Nov 02 23:46:11.338382 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.338331 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" Nov 02 23:46:11.338635 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.338375 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" event={"ID":"195e0f31-6e9c-4861-aa00-2dcb8e5c17de","Type":"ContainerDied","Data":"d0cfedab1e5cf11b0ad5855650a46bee2afbccd9e3adecb378e1a28c4399599f"} Nov 02 23:46:11.338635 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.338419 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" event={"ID":"195e0f31-6e9c-4861-aa00-2dcb8e5c17de","Type":"ContainerDied","Data":"5d5c9ef70f2849f26146de9372e4d0a49fef615727f98ef2633c99877ef4d77b"} Nov 02 23:46:11.338635 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.338439 5114 scope.go:122] "RemoveContainer" containerID="d0cfedab1e5cf11b0ad5855650a46bee2afbccd9e3adecb378e1a28c4399599f" Nov 02 23:46:11.348967 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.348935 5114 scope.go:122] "RemoveContainer" containerID="d0cfedab1e5cf11b0ad5855650a46bee2afbccd9e3adecb378e1a28c4399599f" Nov 02 23:46:11.349306 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:46:11.349288 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0cfedab1e5cf11b0ad5855650a46bee2afbccd9e3adecb378e1a28c4399599f\": container with ID starting with d0cfedab1e5cf11b0ad5855650a46bee2afbccd9e3adecb378e1a28c4399599f not found: ID does not exist" containerID="d0cfedab1e5cf11b0ad5855650a46bee2afbccd9e3adecb378e1a28c4399599f" Nov 02 23:46:11.349346 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.349316 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0cfedab1e5cf11b0ad5855650a46bee2afbccd9e3adecb378e1a28c4399599f"} err="failed to get container status \"d0cfedab1e5cf11b0ad5855650a46bee2afbccd9e3adecb378e1a28c4399599f\": rpc error: code = NotFound desc = could not find container \"d0cfedab1e5cf11b0ad5855650a46bee2afbccd9e3adecb378e1a28c4399599f\": container with ID starting with d0cfedab1e5cf11b0ad5855650a46bee2afbccd9e3adecb378e1a28c4399599f not found: ID does not exist" Nov 02 23:46:11.501793 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.501748 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn"] Nov 02 23:46:11.510881 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.510836 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn"] Nov 02 23:46:11.989817 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:11.989755 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-noten-38049c/noten-model-38049c-kserve-59b767b9cc-h5fzn" podUID="195e0f31-6e9c-4861-aa00-2dcb8e5c17de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:46:12.793073 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:12.793033 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="195e0f31-6e9c-4861-aa00-2dcb8e5c17de" path="/var/lib/kubelet/pods/195e0f31-6e9c-4861-aa00-2dcb8e5c17de/volumes" Nov 02 23:46:13.796800 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:13.796753 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq"] Nov 02 23:46:13.797288 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:13.797245 5114 cpu_manager.go:414] "containerMap: removing container" podUID="195e0f31-6e9c-4861-aa00-2dcb8e5c17de" containerName="main" Nov 02 23:46:13.797288 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:13.797259 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="195e0f31-6e9c-4861-aa00-2dcb8e5c17de" containerName="main" Nov 02 23:46:13.797408 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:13.797382 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="195e0f31-6e9c-4861-aa00-2dcb8e5c17de" containerName="main" Nov 02 23:46:13.801990 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:13.801968 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq" Nov 02 23:46:13.901792 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:13.901680 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b82e0d2a-6469-4f21-966a-a2048217f811-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq\" (UID: \"b82e0d2a-6469-4f21-966a-a2048217f811\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq" Nov 02 23:46:13.902121 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:13.902095 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlrl2\" (UniqueName: \"kubernetes.io/projected/b82e0d2a-6469-4f21-966a-a2048217f811-kube-api-access-hlrl2\") pod \"payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq\" (UID: \"b82e0d2a-6469-4f21-966a-a2048217f811\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq" Nov 02 23:46:13.908279 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:13.905685 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-1498df\"" Nov 02 23:46:13.908464 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:13.908172 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-1498df-dockercfg-jgxdh\"" Nov 02 23:46:13.912847 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:13.912815 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq"] Nov 02 23:46:13.932704 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:13.932669 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg"] Nov 02 23:46:13.936761 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:13.936738 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg" Nov 02 23:46:13.994308 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:13.994271 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg"] Nov 02 23:46:14.010184 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:14.010139 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b82e0d2a-6469-4f21-966a-a2048217f811-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq\" (UID: \"b82e0d2a-6469-4f21-966a-a2048217f811\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq" Nov 02 23:46:14.010343 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:14.010231 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlrl2\" (UniqueName: \"kubernetes.io/projected/b82e0d2a-6469-4f21-966a-a2048217f811-kube-api-access-hlrl2\") pod \"payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq\" (UID: \"b82e0d2a-6469-4f21-966a-a2048217f811\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq" Nov 02 23:46:14.010908 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:14.010886 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b82e0d2a-6469-4f21-966a-a2048217f811-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq\" (UID: \"b82e0d2a-6469-4f21-966a-a2048217f811\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq" Nov 02 23:46:14.071266 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:14.071192 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlrl2\" (UniqueName: \"kubernetes.io/projected/b82e0d2a-6469-4f21-966a-a2048217f811-kube-api-access-hlrl2\") pod \"payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq\" (UID: \"b82e0d2a-6469-4f21-966a-a2048217f811\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq" Nov 02 23:46:14.110976 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:14.110938 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppcvl\" (UniqueName: \"kubernetes.io/projected/b5551d34-16b0-4ed8-97e1-0d49e3d01096-kube-api-access-ppcvl\") pod \"payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg\" (UID: \"b5551d34-16b0-4ed8-97e1-0d49e3d01096\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg" Nov 02 23:46:14.111192 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:14.111059 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5551d34-16b0-4ed8-97e1-0d49e3d01096-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg\" (UID: \"b5551d34-16b0-4ed8-97e1-0d49e3d01096\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg" Nov 02 23:46:14.116836 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:14.116788 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq" Nov 02 23:46:14.214021 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:14.213219 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5551d34-16b0-4ed8-97e1-0d49e3d01096-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg\" (UID: \"b5551d34-16b0-4ed8-97e1-0d49e3d01096\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg" Nov 02 23:46:14.214021 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:14.213424 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppcvl\" (UniqueName: \"kubernetes.io/projected/b5551d34-16b0-4ed8-97e1-0d49e3d01096-kube-api-access-ppcvl\") pod \"payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg\" (UID: \"b5551d34-16b0-4ed8-97e1-0d49e3d01096\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg" Nov 02 23:46:14.214691 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:14.214667 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5551d34-16b0-4ed8-97e1-0d49e3d01096-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg\" (UID: \"b5551d34-16b0-4ed8-97e1-0d49e3d01096\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg" Nov 02 23:46:14.275219 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:14.275177 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppcvl\" (UniqueName: \"kubernetes.io/projected/b5551d34-16b0-4ed8-97e1-0d49e3d01096-kube-api-access-ppcvl\") pod \"payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg\" (UID: \"b5551d34-16b0-4ed8-97e1-0d49e3d01096\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg" Nov 02 23:46:14.501711 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:14.501606 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k"] Nov 02 23:46:14.523693 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:14.523665 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq"] Nov 02 23:46:14.526917 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:46:14.526884 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb82e0d2a_6469_4f21_966a_a2048217f811.slice/crio-387e314403d38449d4358592d25ac29e04fdcd9e807fa805bec4238992f86e91 WatchSource:0}: Error finding container 387e314403d38449d4358592d25ac29e04fdcd9e807fa805bec4238992f86e91: Status 404 returned error can't find the container with id 387e314403d38449d4358592d25ac29e04fdcd9e807fa805bec4238992f86e91 Nov 02 23:46:14.554552 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:14.554516 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg" Nov 02 23:46:14.945448 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:14.945420 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg"] Nov 02 23:46:14.947571 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:46:14.947536 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5551d34_16b0_4ed8_97e1_0d49e3d01096.slice/crio-10165755abfbe038275e844d85c78abb5fac74b45b4a66f2f1f2892349fc8839 WatchSource:0}: Error finding container 10165755abfbe038275e844d85c78abb5fac74b45b4a66f2f1f2892349fc8839: Status 404 returned error can't find the container with id 10165755abfbe038275e844d85c78abb5fac74b45b4a66f2f1f2892349fc8839 Nov 02 23:46:15.365541 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:15.365505 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg" event={"ID":"b5551d34-16b0-4ed8-97e1-0d49e3d01096","Type":"ContainerStarted","Data":"5821b30b00386914ff3b1dc64b9fb4c75b6fca2e7a21b841ea35d4fb3738403f"} Nov 02 23:46:15.365777 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:15.365564 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg" event={"ID":"b5551d34-16b0-4ed8-97e1-0d49e3d01096","Type":"ContainerStarted","Data":"10165755abfbe038275e844d85c78abb5fac74b45b4a66f2f1f2892349fc8839"} Nov 02 23:46:15.365777 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:15.365662 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg" Nov 02 23:46:15.367802 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:15.367769 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq" event={"ID":"b82e0d2a-6469-4f21-966a-a2048217f811","Type":"ContainerStarted","Data":"f4c31ccedbc2ec044cb766737e85e285c5b8c5173036a4512f37f489e6490654"} Nov 02 23:46:15.367980 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:15.367825 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq" event={"ID":"b82e0d2a-6469-4f21-966a-a2048217f811","Type":"ContainerStarted","Data":"387e314403d38449d4358592d25ac29e04fdcd9e807fa805bec4238992f86e91"} Nov 02 23:46:15.367980 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:15.367855 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq" Nov 02 23:46:15.483417 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:15.483358 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg" podStartSLOduration=2.483343242 podStartE2EDuration="2.483343242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:46:15.479784079 +0000 UTC m=+3450.832817455" watchObservedRunningTime="2026-11-02 23:46:15.483343242 +0000 UTC m=+3450.836376610" Nov 02 23:46:15.599549 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:15.599463 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq" podStartSLOduration=2.599441558 podStartE2EDuration="2.599441558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:46:15.596031892 +0000 UTC m=+3450.949065273" watchObservedRunningTime="2026-11-02 23:46:15.599441558 +0000 UTC m=+3450.952474935" Nov 02 23:46:19.502188 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:19.502120 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k" podUID="fcdda5bf-a52f-427a-b68e-9aa605f6682a" containerName="payload-processing" containerID="cri-o://150cc2f3be906200737cb4e0440f0a079ab6f5362023975e88e27b2ab80dc0c3" gracePeriod=25 Nov 02 23:46:19.758464 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:19.758388 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k" Nov 02 23:46:19.783798 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:19.783755 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcdda5bf-a52f-427a-b68e-9aa605f6682a-kube-api-access-pw4wj\" (UniqueName: \"kubernetes.io/projected/fcdda5bf-a52f-427a-b68e-9aa605f6682a-kube-api-access-pw4wj\") pod \"fcdda5bf-a52f-427a-b68e-9aa605f6682a\" (UID: \"fcdda5bf-a52f-427a-b68e-9aa605f6682a\") " Nov 02 23:46:19.783996 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:19.783860 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcdda5bf-a52f-427a-b68e-9aa605f6682a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcdda5bf-a52f-427a-b68e-9aa605f6682a-plugins-config-volume\") pod \"fcdda5bf-a52f-427a-b68e-9aa605f6682a\" (UID: \"fcdda5bf-a52f-427a-b68e-9aa605f6682a\") " Nov 02 23:46:19.784236 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:19.784204 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcdda5bf-a52f-427a-b68e-9aa605f6682a-plugins-config-volume" pod "fcdda5bf-a52f-427a-b68e-9aa605f6682a" (UID: "fcdda5bf-a52f-427a-b68e-9aa605f6682a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:46:19.786142 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:19.786112 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcdda5bf-a52f-427a-b68e-9aa605f6682a-kube-api-access-pw4wj" pod "fcdda5bf-a52f-427a-b68e-9aa605f6682a" (UID: "fcdda5bf-a52f-427a-b68e-9aa605f6682a"). InnerVolumeSpecName "kube-api-access-pw4wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:46:19.886024 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:19.885985 5114 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcdda5bf-a52f-427a-b68e-9aa605f6682a-plugins-config-volume\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:19.886024 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:19.886025 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw4wj\" (UniqueName: \"kubernetes.io/projected/fcdda5bf-a52f-427a-b68e-9aa605f6682a-kube-api-access-pw4wj\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:20.397238 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:20.397199 5114 generic.go:356] "Generic (PLEG): container finished" podID="fcdda5bf-a52f-427a-b68e-9aa605f6682a" containerID="150cc2f3be906200737cb4e0440f0a079ab6f5362023975e88e27b2ab80dc0c3" exitCode=0 Nov 02 23:46:20.397500 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:20.397242 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k" event={"ID":"fcdda5bf-a52f-427a-b68e-9aa605f6682a","Type":"ContainerDied","Data":"150cc2f3be906200737cb4e0440f0a079ab6f5362023975e88e27b2ab80dc0c3"} Nov 02 23:46:20.397680 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:20.397259 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k" Nov 02 23:46:20.397680 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:20.397599 5114 scope.go:122] "RemoveContainer" containerID="150cc2f3be906200737cb4e0440f0a079ab6f5362023975e88e27b2ab80dc0c3" Nov 02 23:46:20.397809 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:20.397573 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k" event={"ID":"fcdda5bf-a52f-427a-b68e-9aa605f6682a","Type":"ContainerDied","Data":"79355e6b02008e5571fa1cb0390f3b3db8d75c77a2c561a465d48924bad1367f"} Nov 02 23:46:20.408530 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:20.408508 5114 scope.go:122] "RemoveContainer" containerID="150cc2f3be906200737cb4e0440f0a079ab6f5362023975e88e27b2ab80dc0c3" Nov 02 23:46:20.408888 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:46:20.408865 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"150cc2f3be906200737cb4e0440f0a079ab6f5362023975e88e27b2ab80dc0c3\": container with ID starting with 150cc2f3be906200737cb4e0440f0a079ab6f5362023975e88e27b2ab80dc0c3 not found: ID does not exist" containerID="150cc2f3be906200737cb4e0440f0a079ab6f5362023975e88e27b2ab80dc0c3" Nov 02 23:46:20.408973 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:20.408898 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"150cc2f3be906200737cb4e0440f0a079ab6f5362023975e88e27b2ab80dc0c3"} err="failed to get container status \"150cc2f3be906200737cb4e0440f0a079ab6f5362023975e88e27b2ab80dc0c3\": rpc error: code = NotFound desc = could not find container \"150cc2f3be906200737cb4e0440f0a079ab6f5362023975e88e27b2ab80dc0c3\": container with ID starting with 150cc2f3be906200737cb4e0440f0a079ab6f5362023975e88e27b2ab80dc0c3 not found: ID does not exist" Nov 02 23:46:20.559975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:20.559944 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k"] Nov 02 23:46:20.609471 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:20.609434 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-875049-5b7f7c74b7-p8z5k"] Nov 02 23:46:20.793313 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:20.793272 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcdda5bf-a52f-427a-b68e-9aa605f6682a" path="/var/lib/kubelet/pods/fcdda5bf-a52f-427a-b68e-9aa605f6682a/volumes" Nov 02 23:46:23.264270 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.264228 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7"] Nov 02 23:46:23.264856 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.264816 5114 cpu_manager.go:414] "containerMap: removing container" podUID="fcdda5bf-a52f-427a-b68e-9aa605f6682a" containerName="payload-processing" Nov 02 23:46:23.264856 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.264836 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcdda5bf-a52f-427a-b68e-9aa605f6682a" containerName="payload-processing" Nov 02 23:46:23.264981 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.264972 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcdda5bf-a52f-427a-b68e-9aa605f6682a" containerName="payload-processing" Nov 02 23:46:23.269508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.269470 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.344318 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.330190 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-workload-certs\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.344318 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.330276 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-credential-socket\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.344318 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.330349 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7sml\" (UniqueName: \"kubernetes.io/projected/6f16d749-1be4-4d77-bb08-21bcf16bb724-kube-api-access-l7sml\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.344318 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.330409 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-workload-socket\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.344318 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.330468 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-token\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.344318 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.330514 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-podinfo\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.344318 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.330543 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f16d749-1be4-4d77-bb08-21bcf16bb724-istiod-ca-cert\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.344318 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.330582 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-envoy\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.344318 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.330611 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-data\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.360869 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.360829 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7"] Nov 02 23:46:23.400235 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.400196 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-6c3d8e-openshift-default-dockercfg-xf996\"" Nov 02 23:46:23.432184 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.431219 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-token\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.432184 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.431278 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-podinfo\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.432184 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.431313 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f16d749-1be4-4d77-bb08-21bcf16bb724-istiod-ca-cert\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.432184 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.431357 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-envoy\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.432184 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.431392 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-data\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.432184 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.431442 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-workload-certs\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.433094 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.433068 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-credential-socket\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.433228 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.433181 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7sml\" (UniqueName: \"kubernetes.io/projected/6f16d749-1be4-4d77-bb08-21bcf16bb724-kube-api-access-l7sml\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.433296 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.433255 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-workload-socket\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.433747 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.433693 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-data\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.434049 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.433981 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-workload-certs\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.434661 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.434520 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f16d749-1be4-4d77-bb08-21bcf16bb724-istiod-ca-cert\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.434835 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.434760 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-credential-socket\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.435067 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.435045 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-workload-socket\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.437700 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.437678 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-podinfo\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.448233 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.448193 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-envoy\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.493032 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.492993 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-token\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.493422 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.493401 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7sml\" (UniqueName: \"kubernetes.io/projected/6f16d749-1be4-4d77-bb08-21bcf16bb724-kube-api-access-l7sml\") pod \"e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:23.584044 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:23.584009 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:24.024012 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:24.023909 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7"] Nov 02 23:46:24.030339 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:46:24.030295 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f16d749_1be4_4d77_bb08_21bcf16bb724.slice/crio-d1aba4cde2610319214e1844b4010a3fb4198883b68c2a3e9b63d8d5b095392f WatchSource:0}: Error finding container d1aba4cde2610319214e1844b4010a3fb4198883b68c2a3e9b63d8d5b095392f: Status 404 returned error can't find the container with id d1aba4cde2610319214e1844b4010a3fb4198883b68c2a3e9b63d8d5b095392f Nov 02 23:46:24.033323 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:24.033285 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:46:24.033446 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:24.033363 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:46:24.033446 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:24.033395 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:46:24.433888 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:24.433846 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" event={"ID":"6f16d749-1be4-4d77-bb08-21bcf16bb724","Type":"ContainerStarted","Data":"989320271dcb5483ab9e258b6f1ef2e26f3f2f69393dac19dbc9ffe21ec60ff4"} Nov 02 23:46:24.434325 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:24.433913 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" event={"ID":"6f16d749-1be4-4d77-bb08-21bcf16bb724","Type":"ContainerStarted","Data":"d1aba4cde2610319214e1844b4010a3fb4198883b68c2a3e9b63d8d5b095392f"} Nov 02 23:46:24.556134 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:24.556063 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" podStartSLOduration=1.556041139 podStartE2EDuration="1.556041139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:46:24.553287654 +0000 UTC m=+3459.906321034" watchObservedRunningTime="2026-11-02 23:46:24.556041139 +0000 UTC m=+3459.909074515" Nov 02 23:46:24.584432 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:24.584393 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:25.590925 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:25.590893 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:26.377136 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:26.377105 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq" Nov 02 23:46:26.377366 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:26.377178 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg" Nov 02 23:46:26.445212 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:26.445176 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:26.446157 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:26.446134 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:46:30.960738 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:30.960694 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8"] Nov 02 23:46:30.965768 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:30.965738 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8" Nov 02 23:46:31.022800 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:31.022755 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8"] Nov 02 23:46:31.031917 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:31.031876 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f278fc6e-68f2-46d0-b405-76993550b9f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8\" (UID: \"f278fc6e-68f2-46d0-b405-76993550b9f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8" Nov 02 23:46:31.032514 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:31.032462 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4w8v\" (UniqueName: \"kubernetes.io/projected/f278fc6e-68f2-46d0-b405-76993550b9f9-kube-api-access-n4w8v\") pod \"payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8\" (UID: \"f278fc6e-68f2-46d0-b405-76993550b9f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8" Nov 02 23:46:31.069753 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:31.069723 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-6c3d8e\"" Nov 02 23:46:31.070392 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:31.070156 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-6c3d8e-dockercfg-rsjhf\"" Nov 02 23:46:31.122012 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:31.121974 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p"] Nov 02 23:46:31.125999 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:31.125971 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p" Nov 02 23:46:31.134204 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:31.134177 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4w8v\" (UniqueName: \"kubernetes.io/projected/f278fc6e-68f2-46d0-b405-76993550b9f9-kube-api-access-n4w8v\") pod \"payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8\" (UID: \"f278fc6e-68f2-46d0-b405-76993550b9f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8" Nov 02 23:46:31.134301 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:31.134219 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f278fc6e-68f2-46d0-b405-76993550b9f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8\" (UID: \"f278fc6e-68f2-46d0-b405-76993550b9f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8" Nov 02 23:46:31.134918 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:31.134897 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f278fc6e-68f2-46d0-b405-76993550b9f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8\" (UID: \"f278fc6e-68f2-46d0-b405-76993550b9f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8" Nov 02 23:46:31.174306 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:31.174268 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p"] Nov 02 23:46:31.195662 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:31.195625 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4w8v\" (UniqueName: \"kubernetes.io/projected/f278fc6e-68f2-46d0-b405-76993550b9f9-kube-api-access-n4w8v\") pod \"payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8\" (UID: \"f278fc6e-68f2-46d0-b405-76993550b9f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8" Nov 02 23:46:31.235396 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:31.235297 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q24h\" (UniqueName: \"kubernetes.io/projected/5d1a1875-76ae-454b-9d4c-ee871f1d6a22-kube-api-access-7q24h\") pod \"payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p\" (UID: \"5d1a1875-76ae-454b-9d4c-ee871f1d6a22\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p" Nov 02 23:46:31.235599 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:31.235535 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d1a1875-76ae-454b-9d4c-ee871f1d6a22-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p\" (UID: \"5d1a1875-76ae-454b-9d4c-ee871f1d6a22\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p" Nov 02 23:46:31.280145 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:31.280110 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8" Nov 02 23:46:31.337936 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:31.337873 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d1a1875-76ae-454b-9d4c-ee871f1d6a22-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p\" (UID: \"5d1a1875-76ae-454b-9d4c-ee871f1d6a22\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p" Nov 02 23:46:31.338078 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:31.337990 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q24h\" (UniqueName: \"kubernetes.io/projected/5d1a1875-76ae-454b-9d4c-ee871f1d6a22-kube-api-access-7q24h\") pod \"payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p\" (UID: \"5d1a1875-76ae-454b-9d4c-ee871f1d6a22\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p" Nov 02 23:46:31.338910 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:31.338888 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d1a1875-76ae-454b-9d4c-ee871f1d6a22-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p\" (UID: \"5d1a1875-76ae-454b-9d4c-ee871f1d6a22\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p" Nov 02 23:46:31.399440 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:31.399404 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q24h\" (UniqueName: \"kubernetes.io/projected/5d1a1875-76ae-454b-9d4c-ee871f1d6a22-kube-api-access-7q24h\") pod \"payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p\" (UID: \"5d1a1875-76ae-454b-9d4c-ee871f1d6a22\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p" Nov 02 23:46:31.439763 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:31.439728 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p" Nov 02 23:46:31.656254 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:31.656230 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8"] Nov 02 23:46:31.658002 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:46:31.657970 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf278fc6e_68f2_46d0_b405_76993550b9f9.slice/crio-547b768514a95d5199cd78198f1829037182326267e5273263a3ef4692125512 WatchSource:0}: Error finding container 547b768514a95d5199cd78198f1829037182326267e5273263a3ef4692125512: Status 404 returned error can't find the container with id 547b768514a95d5199cd78198f1829037182326267e5273263a3ef4692125512 Nov 02 23:46:31.818616 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:31.818582 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p"] Nov 02 23:46:31.820617 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:46:31.820591 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d1a1875_76ae_454b_9d4c_ee871f1d6a22.slice/crio-3da3fed525819f95e72c4563e113cffee1b948fb4705e6ed0bc8f1561c08ff71 WatchSource:0}: Error finding container 3da3fed525819f95e72c4563e113cffee1b948fb4705e6ed0bc8f1561c08ff71: Status 404 returned error can't find the container with id 3da3fed525819f95e72c4563e113cffee1b948fb4705e6ed0bc8f1561c08ff71 Nov 02 23:46:32.481826 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:32.481785 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p" event={"ID":"5d1a1875-76ae-454b-9d4c-ee871f1d6a22","Type":"ContainerStarted","Data":"ab99d54b354592a521dea5ffa6c5f8f3981ca9838427afea04ae8288a2a8ee82"} Nov 02 23:46:32.482389 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:32.481844 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p" event={"ID":"5d1a1875-76ae-454b-9d4c-ee871f1d6a22","Type":"ContainerStarted","Data":"3da3fed525819f95e72c4563e113cffee1b948fb4705e6ed0bc8f1561c08ff71"} Nov 02 23:46:32.482389 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:32.481872 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p" Nov 02 23:46:32.483650 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:32.483618 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8" event={"ID":"f278fc6e-68f2-46d0-b405-76993550b9f9","Type":"ContainerStarted","Data":"176e506aa325ecd25b177a3c338ce3d8c63696b9eef1c1af925e0bc719c910da"} Nov 02 23:46:32.483794 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:32.483659 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8" event={"ID":"f278fc6e-68f2-46d0-b405-76993550b9f9","Type":"ContainerStarted","Data":"547b768514a95d5199cd78198f1829037182326267e5273263a3ef4692125512"} Nov 02 23:46:32.483794 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:32.483697 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8" Nov 02 23:46:32.601618 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:32.601553 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p" podStartSLOduration=1.601536691 podStartE2EDuration="1.601536691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:46:32.598544336 +0000 UTC m=+3467.951577711" watchObservedRunningTime="2026-11-02 23:46:32.601536691 +0000 UTC m=+3467.954570066" Nov 02 23:46:32.715259 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:32.715175 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8" podStartSLOduration=2.71511698 podStartE2EDuration="2.71511698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:46:32.712983144 +0000 UTC m=+3468.066016519" watchObservedRunningTime="2026-11-02 23:46:32.71511698 +0000 UTC m=+3468.068150358" Nov 02 23:46:37.589259 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:46:37.589220 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:46:41.273204 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:41.273165 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst"] Nov 02 23:46:41.273742 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:41.273403 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" podUID="fa812057-3300-4df2-acf5-59ac9fd4999e" containerName="istio-proxy" containerID="cri-o://0465d92a44345b389d3f81bb52d879e6ebc50d222ce1605fd762c2f63c95fe3b" gracePeriod=30 Nov 02 23:46:43.050938 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:43.050896 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc"] Nov 02 23:46:43.056010 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:43.055982 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc" Nov 02 23:46:43.077767 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:43.077731 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d286a90d-42ec-4b33-8180-1a6afcc1b3ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc\" (UID: \"d286a90d-42ec-4b33-8180-1a6afcc1b3ee\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc" Nov 02 23:46:43.077953 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:43.077874 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w56pq\" (UniqueName: \"kubernetes.io/projected/d286a90d-42ec-4b33-8180-1a6afcc1b3ee-kube-api-access-w56pq\") pod \"payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc\" (UID: \"d286a90d-42ec-4b33-8180-1a6afcc1b3ee\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc" Nov 02 23:46:43.113428 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:43.113382 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc"] Nov 02 23:46:43.159096 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:43.159056 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-0f49ab\"" Nov 02 23:46:43.179096 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:43.179050 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d286a90d-42ec-4b33-8180-1a6afcc1b3ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc\" (UID: \"d286a90d-42ec-4b33-8180-1a6afcc1b3ee\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc" Nov 02 23:46:43.179571 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:43.179542 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w56pq\" (UniqueName: \"kubernetes.io/projected/d286a90d-42ec-4b33-8180-1a6afcc1b3ee-kube-api-access-w56pq\") pod \"payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc\" (UID: \"d286a90d-42ec-4b33-8180-1a6afcc1b3ee\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc" Nov 02 23:46:43.180519 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:43.180467 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d286a90d-42ec-4b33-8180-1a6afcc1b3ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc\" (UID: \"d286a90d-42ec-4b33-8180-1a6afcc1b3ee\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc" Nov 02 23:46:43.240403 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:43.240372 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w56pq\" (UniqueName: \"kubernetes.io/projected/d286a90d-42ec-4b33-8180-1a6afcc1b3ee-kube-api-access-w56pq\") pod \"payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc\" (UID: \"d286a90d-42ec-4b33-8180-1a6afcc1b3ee\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc" Nov 02 23:46:43.371249 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:43.371151 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc" Nov 02 23:46:43.492675 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:43.492645 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8" Nov 02 23:46:43.492887 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:43.492737 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p" Nov 02 23:46:43.750639 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:43.750613 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc"] Nov 02 23:46:43.753037 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:46:43.753000 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd286a90d_42ec_4b33_8180_1a6afcc1b3ee.slice/crio-6153aa3a135ab22907b9bbda06553a425f614f45ba3bea48cdf04c3d57292064 WatchSource:0}: Error finding container 6153aa3a135ab22907b9bbda06553a425f614f45ba3bea48cdf04c3d57292064: Status 404 returned error can't find the container with id 6153aa3a135ab22907b9bbda06553a425f614f45ba3bea48cdf04c3d57292064 Nov 02 23:46:44.558814 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:44.558774 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc" event={"ID":"d286a90d-42ec-4b33-8180-1a6afcc1b3ee","Type":"ContainerStarted","Data":"2d70292124389839d322e4f632fe3ba2d72f9b607152d80cbd6d2e9d0a4860d4"} Nov 02 23:46:44.559314 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:44.558842 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc" event={"ID":"d286a90d-42ec-4b33-8180-1a6afcc1b3ee","Type":"ContainerStarted","Data":"6153aa3a135ab22907b9bbda06553a425f614f45ba3bea48cdf04c3d57292064"} Nov 02 23:46:44.559314 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:44.558897 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc" Nov 02 23:46:44.675736 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:44.675670 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc" podStartSLOduration=1.675640204 podStartE2EDuration="1.675640204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:46:44.674933066 +0000 UTC m=+3480.027966442" watchObservedRunningTime="2026-11-02 23:46:44.675640204 +0000 UTC m=+3480.028673579" Nov 02 23:46:46.549079 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.548990 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:46:46.575043 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.575002 5114 generic.go:356] "Generic (PLEG): container finished" podID="fa812057-3300-4df2-acf5-59ac9fd4999e" containerID="0465d92a44345b389d3f81bb52d879e6ebc50d222ce1605fd762c2f63c95fe3b" exitCode=0 Nov 02 23:46:46.575215 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.575055 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" Nov 02 23:46:46.575215 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.575083 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" event={"ID":"fa812057-3300-4df2-acf5-59ac9fd4999e","Type":"ContainerDied","Data":"0465d92a44345b389d3f81bb52d879e6ebc50d222ce1605fd762c2f63c95fe3b"} Nov 02 23:46:46.575215 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.575140 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst" event={"ID":"fa812057-3300-4df2-acf5-59ac9fd4999e","Type":"ContainerDied","Data":"22e2683cc8e2889a46e31239dee05cbf238d3b3514cecbae4cc34c84df04bd2e"} Nov 02 23:46:46.575215 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.575171 5114 scope.go:122] "RemoveContainer" containerID="0465d92a44345b389d3f81bb52d879e6ebc50d222ce1605fd762c2f63c95fe3b" Nov 02 23:46:46.584786 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.584767 5114 scope.go:122] "RemoveContainer" containerID="0465d92a44345b389d3f81bb52d879e6ebc50d222ce1605fd762c2f63c95fe3b" Nov 02 23:46:46.585072 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:46:46.585054 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0465d92a44345b389d3f81bb52d879e6ebc50d222ce1605fd762c2f63c95fe3b\": container with ID starting with 0465d92a44345b389d3f81bb52d879e6ebc50d222ce1605fd762c2f63c95fe3b not found: ID does not exist" containerID="0465d92a44345b389d3f81bb52d879e6ebc50d222ce1605fd762c2f63c95fe3b" Nov 02 23:46:46.585117 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.585085 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0465d92a44345b389d3f81bb52d879e6ebc50d222ce1605fd762c2f63c95fe3b"} err="failed to get container status \"0465d92a44345b389d3f81bb52d879e6ebc50d222ce1605fd762c2f63c95fe3b\": rpc error: code = NotFound desc = could not find container \"0465d92a44345b389d3f81bb52d879e6ebc50d222ce1605fd762c2f63c95fe3b\": container with ID starting with 0465d92a44345b389d3f81bb52d879e6ebc50d222ce1605fd762c2f63c95fe3b not found: ID does not exist" Nov 02 23:46:46.619896 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.619856 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-podinfo\") pod \"fa812057-3300-4df2-acf5-59ac9fd4999e\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " Nov 02 23:46:46.619896 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.619905 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-envoy\") pod \"fa812057-3300-4df2-acf5-59ac9fd4999e\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " Nov 02 23:46:46.620148 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.619937 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-workload-certs\") pod \"fa812057-3300-4df2-acf5-59ac9fd4999e\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " Nov 02 23:46:46.620148 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.620041 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-workload-socket\") pod \"fa812057-3300-4df2-acf5-59ac9fd4999e\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " Nov 02 23:46:46.620148 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.620074 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa812057-3300-4df2-acf5-59ac9fd4999e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa812057-3300-4df2-acf5-59ac9fd4999e-istiod-ca-cert\") pod \"fa812057-3300-4df2-acf5-59ac9fd4999e\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " Nov 02 23:46:46.620324 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.620189 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-data\") pod \"fa812057-3300-4df2-acf5-59ac9fd4999e\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " Nov 02 23:46:46.620324 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.620234 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-token\" (UniqueName: \"kubernetes.io/projected/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-token\") pod \"fa812057-3300-4df2-acf5-59ac9fd4999e\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " Nov 02 23:46:46.620324 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.620249 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-workload-certs" pod "fa812057-3300-4df2-acf5-59ac9fd4999e" (UID: "fa812057-3300-4df2-acf5-59ac9fd4999e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:46.620324 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.620287 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-credential-socket\") pod \"fa812057-3300-4df2-acf5-59ac9fd4999e\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " Nov 02 23:46:46.620324 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.620318 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa812057-3300-4df2-acf5-59ac9fd4999e-kube-api-access-vb85v\" (UniqueName: \"kubernetes.io/projected/fa812057-3300-4df2-acf5-59ac9fd4999e-kube-api-access-vb85v\") pod \"fa812057-3300-4df2-acf5-59ac9fd4999e\" (UID: \"fa812057-3300-4df2-acf5-59ac9fd4999e\") " Nov 02 23:46:46.620638 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.620328 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-workload-socket" pod "fa812057-3300-4df2-acf5-59ac9fd4999e" (UID: "fa812057-3300-4df2-acf5-59ac9fd4999e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:46.620638 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.620567 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-data" pod "fa812057-3300-4df2-acf5-59ac9fd4999e" (UID: "fa812057-3300-4df2-acf5-59ac9fd4999e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:46.620638 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.620617 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-credential-socket" pod "fa812057-3300-4df2-acf5-59ac9fd4999e" (UID: "fa812057-3300-4df2-acf5-59ac9fd4999e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:46.620784 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.620661 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa812057-3300-4df2-acf5-59ac9fd4999e-istiod-ca-cert" pod "fa812057-3300-4df2-acf5-59ac9fd4999e" (UID: "fa812057-3300-4df2-acf5-59ac9fd4999e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:46:46.620923 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.620899 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:46.620923 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.620924 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:46.621097 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.620938 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa812057-3300-4df2-acf5-59ac9fd4999e-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:46.621097 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.620951 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:46.621097 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.620970 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:46.622439 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.622412 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-envoy" pod "fa812057-3300-4df2-acf5-59ac9fd4999e" (UID: "fa812057-3300-4df2-acf5-59ac9fd4999e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:46.622669 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.622649 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa812057-3300-4df2-acf5-59ac9fd4999e-kube-api-access-vb85v" pod "fa812057-3300-4df2-acf5-59ac9fd4999e" (UID: "fa812057-3300-4df2-acf5-59ac9fd4999e"). InnerVolumeSpecName "kube-api-access-vb85v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:46:46.622954 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.622932 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-podinfo" pod "fa812057-3300-4df2-acf5-59ac9fd4999e" (UID: "fa812057-3300-4df2-acf5-59ac9fd4999e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:46:46.623016 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.622950 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-token" pod "fa812057-3300-4df2-acf5-59ac9fd4999e" (UID: "fa812057-3300-4df2-acf5-59ac9fd4999e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:46:46.721750 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.721717 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:46.721750 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.721746 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:46.721750 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.721755 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa812057-3300-4df2-acf5-59ac9fd4999e-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:46.722020 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.721764 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb85v\" (UniqueName: \"kubernetes.io/projected/fa812057-3300-4df2-acf5-59ac9fd4999e-kube-api-access-vb85v\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:46.989435 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.989399 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst"] Nov 02 23:46:46.993131 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:46.993100 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-npfst"] Nov 02 23:46:48.794017 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:48.793982 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa812057-3300-4df2-acf5-59ac9fd4999e" path="/var/lib/kubelet/pods/fa812057-3300-4df2-acf5-59ac9fd4999e/volumes" Nov 02 23:46:55.567954 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:46:55.567924 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc" Nov 02 23:47:02.424752 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:02.424717 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-875049-lbb59"] Nov 02 23:47:02.425406 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:02.425381 5114 cpu_manager.go:414] "containerMap: removing container" podUID="fa812057-3300-4df2-acf5-59ac9fd4999e" containerName="istio-proxy" Nov 02 23:47:02.425541 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:02.425407 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa812057-3300-4df2-acf5-59ac9fd4999e" containerName="istio-proxy" Nov 02 23:47:02.425628 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:02.425611 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa812057-3300-4df2-acf5-59ac9fd4999e" containerName="istio-proxy" Nov 02 23:47:02.430184 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:02.430156 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-875049-lbb59" Nov 02 23:47:02.482719 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:02.482681 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:47:02.532079 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:02.532049 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:47:02.534464 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:02.534438 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-875049-lbb59"] Nov 02 23:47:02.608999 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:02.608948 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3764839-17c4-47af-b560-91d4bb3b06f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-875049-lbb59\" (UID: \"f3764839-17c4-47af-b560-91d4bb3b06f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-875049-lbb59" Nov 02 23:47:02.609180 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:02.609062 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96wg4\" (UniqueName: \"kubernetes.io/projected/f3764839-17c4-47af-b560-91d4bb3b06f1-kube-api-access-96wg4\") pod \"maas-api-revoke-keys-e2e-autores-875049-lbb59\" (UID: \"f3764839-17c4-47af-b560-91d4bb3b06f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-875049-lbb59" Nov 02 23:47:02.609180 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:02.609101 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3764839-17c4-47af-b560-91d4bb3b06f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-875049-lbb59\" (UID: \"f3764839-17c4-47af-b560-91d4bb3b06f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-875049-lbb59" Nov 02 23:47:02.710392 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:02.710304 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3764839-17c4-47af-b560-91d4bb3b06f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-875049-lbb59\" (UID: \"f3764839-17c4-47af-b560-91d4bb3b06f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-875049-lbb59" Nov 02 23:47:02.710392 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:02.710356 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96wg4\" (UniqueName: \"kubernetes.io/projected/f3764839-17c4-47af-b560-91d4bb3b06f1-kube-api-access-96wg4\") pod \"maas-api-revoke-keys-e2e-autores-875049-lbb59\" (UID: \"f3764839-17c4-47af-b560-91d4bb3b06f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-875049-lbb59" Nov 02 23:47:02.710628 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:02.710394 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3764839-17c4-47af-b560-91d4bb3b06f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-875049-lbb59\" (UID: \"f3764839-17c4-47af-b560-91d4bb3b06f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-875049-lbb59" Nov 02 23:47:02.711191 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:02.711169 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3764839-17c4-47af-b560-91d4bb3b06f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-875049-lbb59\" (UID: \"f3764839-17c4-47af-b560-91d4bb3b06f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-875049-lbb59" Nov 02 23:47:02.713044 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:02.713021 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3764839-17c4-47af-b560-91d4bb3b06f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-875049-lbb59\" (UID: \"f3764839-17c4-47af-b560-91d4bb3b06f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-875049-lbb59" Nov 02 23:47:02.771138 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:02.771096 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96wg4\" (UniqueName: \"kubernetes.io/projected/f3764839-17c4-47af-b560-91d4bb3b06f1-kube-api-access-96wg4\") pod \"maas-api-revoke-keys-e2e-autores-875049-lbb59\" (UID: \"f3764839-17c4-47af-b560-91d4bb3b06f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-875049-lbb59" Nov 02 23:47:03.044039 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:03.044002 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-875049-lbb59" Nov 02 23:47:03.422244 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:03.422207 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-875049-lbb59"] Nov 02 23:47:03.423708 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:47:03.423678 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3764839_17c4_47af_b560_91d4bb3b06f1.slice/crio-2dc5f52d2886caa1e8617bfbdaac27aea2098fc0c8c4dd1234bf3e6fc55af56f WatchSource:0}: Error finding container 2dc5f52d2886caa1e8617bfbdaac27aea2098fc0c8c4dd1234bf3e6fc55af56f: Status 404 returned error can't find the container with id 2dc5f52d2886caa1e8617bfbdaac27aea2098fc0c8c4dd1234bf3e6fc55af56f Nov 02 23:47:03.683091 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:03.683009 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-875049-lbb59" event={"ID":"f3764839-17c4-47af-b560-91d4bb3b06f1","Type":"ContainerStarted","Data":"2dc5f52d2886caa1e8617bfbdaac27aea2098fc0c8c4dd1234bf3e6fc55af56f"} Nov 02 23:47:04.689892 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:04.689849 5114 generic.go:356] "Generic (PLEG): container finished" podID="f3764839-17c4-47af-b560-91d4bb3b06f1" containerID="fc245d2caaf0f037230538c9d8d2279e4659dfaa2717bde9a684b8efeb6526ec" exitCode=0 Nov 02 23:47:04.690371 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:04.689923 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-875049-lbb59" event={"ID":"f3764839-17c4-47af-b560-91d4bb3b06f1","Type":"ContainerDied","Data":"fc245d2caaf0f037230538c9d8d2279e4659dfaa2717bde9a684b8efeb6526ec"} Nov 02 23:47:05.849537 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:05.849510 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-875049-lbb59" Nov 02 23:47:05.854862 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:05.854826 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3764839-17c4-47af-b560-91d4bb3b06f1-kube-api-access-96wg4\" (UniqueName: \"kubernetes.io/projected/f3764839-17c4-47af-b560-91d4bb3b06f1-kube-api-access-96wg4\") pod \"f3764839-17c4-47af-b560-91d4bb3b06f1\" (UID: \"f3764839-17c4-47af-b560-91d4bb3b06f1\") " Nov 02 23:47:05.855025 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:05.854960 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3764839-17c4-47af-b560-91d4bb3b06f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3764839-17c4-47af-b560-91d4bb3b06f1-maas-api-openshift-ca\") pod \"f3764839-17c4-47af-b560-91d4bb3b06f1\" (UID: \"f3764839-17c4-47af-b560-91d4bb3b06f1\") " Nov 02 23:47:05.855025 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:05.855004 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3764839-17c4-47af-b560-91d4bb3b06f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3764839-17c4-47af-b560-91d4bb3b06f1-maas-api-xks-ca\") pod \"f3764839-17c4-47af-b560-91d4bb3b06f1\" (UID: \"f3764839-17c4-47af-b560-91d4bb3b06f1\") " Nov 02 23:47:05.855390 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:05.855368 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3764839-17c4-47af-b560-91d4bb3b06f1-maas-api-openshift-ca" pod "f3764839-17c4-47af-b560-91d4bb3b06f1" (UID: "f3764839-17c4-47af-b560-91d4bb3b06f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:47:05.857237 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:05.857204 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3764839-17c4-47af-b560-91d4bb3b06f1-maas-api-xks-ca" pod "f3764839-17c4-47af-b560-91d4bb3b06f1" (UID: "f3764839-17c4-47af-b560-91d4bb3b06f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:47:05.857237 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:05.857230 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3764839-17c4-47af-b560-91d4bb3b06f1-kube-api-access-96wg4" pod "f3764839-17c4-47af-b560-91d4bb3b06f1" (UID: "f3764839-17c4-47af-b560-91d4bb3b06f1"). InnerVolumeSpecName "kube-api-access-96wg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:47:05.956058 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:05.955970 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96wg4\" (UniqueName: \"kubernetes.io/projected/f3764839-17c4-47af-b560-91d4bb3b06f1-kube-api-access-96wg4\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:47:05.956058 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:05.956001 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3764839-17c4-47af-b560-91d4bb3b06f1-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:47:05.956058 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:05.956014 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3764839-17c4-47af-b560-91d4bb3b06f1-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:47:06.703324 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:06.703284 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-875049-lbb59" Nov 02 23:47:06.703552 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:06.703306 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-875049-lbb59" event={"ID":"f3764839-17c4-47af-b560-91d4bb3b06f1","Type":"ContainerDied","Data":"2dc5f52d2886caa1e8617bfbdaac27aea2098fc0c8c4dd1234bf3e6fc55af56f"} Nov 02 23:47:06.703552 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:06.703399 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dc5f52d2886caa1e8617bfbdaac27aea2098fc0c8c4dd1234bf3e6fc55af56f" Nov 02 23:47:07.584360 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:47:07.584323 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:47:37.584426 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:47:37.584388 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:47:43.205015 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:43.204965 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv"] Nov 02 23:47:43.205679 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:43.205656 5114 cpu_manager.go:414] "containerMap: removing container" podUID="f3764839-17c4-47af-b560-91d4bb3b06f1" containerName="revoke-keys" Nov 02 23:47:43.205784 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:43.205681 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3764839-17c4-47af-b560-91d4bb3b06f1" containerName="revoke-keys" Nov 02 23:47:43.205853 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:43.205831 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3764839-17c4-47af-b560-91d4bb3b06f1" containerName="revoke-keys" Nov 02 23:47:43.209504 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:43.209464 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" Nov 02 23:47:43.261540 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:43.261468 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-6c3d8e-serving-cert\"" Nov 02 23:47:43.278185 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:43.278136 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x82s\" (UniqueName: \"kubernetes.io/projected/dc281e12-676c-41ec-9cc9-ac10fab03e26-kube-api-access-4x82s\") pod \"maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv\" (UID: \"dc281e12-676c-41ec-9cc9-ac10fab03e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" Nov 02 23:47:43.278471 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:43.278198 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc281e12-676c-41ec-9cc9-ac10fab03e26-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv\" (UID: \"dc281e12-676c-41ec-9cc9-ac10fab03e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" Nov 02 23:47:43.278471 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:43.278355 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc281e12-676c-41ec-9cc9-ac10fab03e26-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv\" (UID: \"dc281e12-676c-41ec-9cc9-ac10fab03e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" Nov 02 23:47:43.278471 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:43.278408 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc281e12-676c-41ec-9cc9-ac10fab03e26-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv\" (UID: \"dc281e12-676c-41ec-9cc9-ac10fab03e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" Nov 02 23:47:43.313790 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:43.313746 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv"] Nov 02 23:47:43.379653 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:43.379614 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc281e12-676c-41ec-9cc9-ac10fab03e26-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv\" (UID: \"dc281e12-676c-41ec-9cc9-ac10fab03e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" Nov 02 23:47:43.379880 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:43.379670 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc281e12-676c-41ec-9cc9-ac10fab03e26-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv\" (UID: \"dc281e12-676c-41ec-9cc9-ac10fab03e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" Nov 02 23:47:43.379880 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:43.379769 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x82s\" (UniqueName: \"kubernetes.io/projected/dc281e12-676c-41ec-9cc9-ac10fab03e26-kube-api-access-4x82s\") pod \"maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv\" (UID: \"dc281e12-676c-41ec-9cc9-ac10fab03e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" Nov 02 23:47:43.379880 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:43.379791 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc281e12-676c-41ec-9cc9-ac10fab03e26-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv\" (UID: \"dc281e12-676c-41ec-9cc9-ac10fab03e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" Nov 02 23:47:43.379880 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:47:43.379822 5114 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-serving-cert: secret "maas-api-e2e-shared-b-w4-6c3d8e-serving-cert" not found Nov 02 23:47:43.380104 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:47:43.379902 5114 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc281e12-676c-41ec-9cc9-ac10fab03e26-maas-api-tls podName:dc281e12-676c-41ec-9cc9-ac10fab03e26 nodeName:}" failed. No retries permitted until 2026-11-02 23:47:43.879882979 +0000 UTC m=+3539.232916348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc281e12-676c-41ec-9cc9-ac10fab03e26-maas-api-tls") pod "maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" (UID: "dc281e12-676c-41ec-9cc9-ac10fab03e26") : secret "maas-api-e2e-shared-b-w4-6c3d8e-serving-cert" not found Nov 02 23:47:43.380431 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:43.380412 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc281e12-676c-41ec-9cc9-ac10fab03e26-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv\" (UID: \"dc281e12-676c-41ec-9cc9-ac10fab03e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" Nov 02 23:47:43.382350 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:43.382324 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc281e12-676c-41ec-9cc9-ac10fab03e26-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv\" (UID: \"dc281e12-676c-41ec-9cc9-ac10fab03e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" Nov 02 23:47:43.441231 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:43.441199 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x82s\" (UniqueName: \"kubernetes.io/projected/dc281e12-676c-41ec-9cc9-ac10fab03e26-kube-api-access-4x82s\") pod \"maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv\" (UID: \"dc281e12-676c-41ec-9cc9-ac10fab03e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" Nov 02 23:47:43.884116 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:43.884071 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc281e12-676c-41ec-9cc9-ac10fab03e26-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv\" (UID: \"dc281e12-676c-41ec-9cc9-ac10fab03e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" Nov 02 23:47:43.886844 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:43.886804 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc281e12-676c-41ec-9cc9-ac10fab03e26-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv\" (UID: \"dc281e12-676c-41ec-9cc9-ac10fab03e26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" Nov 02 23:47:44.123630 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:44.123591 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" Nov 02 23:47:44.507105 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:44.503880 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv"] Nov 02 23:47:44.510094 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:44.510067 5114 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:47:44.925716 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:44.925675 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" event={"ID":"dc281e12-676c-41ec-9cc9-ac10fab03e26","Type":"ContainerStarted","Data":"3b5b8815d6071ae7bbd7afc9f2bfa0ba7c23f6916e820fe2e1808b21542e1be8"} Nov 02 23:47:45.793822 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:45.793783 5114 scope.go:122] "RemoveContainer" containerID="2cec59850e5a59806f801c9ba85162e76c794cab7a2c913048163923cfa6d454" Nov 02 23:47:45.937272 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:45.937233 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" event={"ID":"dc281e12-676c-41ec-9cc9-ac10fab03e26","Type":"ContainerStarted","Data":"7c7d2ba603d7bca06c7454a0fb7c50932b1530a920aca8d9a1d3f46a37f26b68"} Nov 02 23:47:45.937498 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:45.937432 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" Nov 02 23:47:46.056941 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:46.056833 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" podStartSLOduration=2.211424207 podStartE2EDuration="3.056818958s" totalImagesPullingTime="845.394751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:47:46.055289675 +0000 UTC m=+3541.408323051" watchObservedRunningTime="2026-11-02 23:47:46.056818958 +0000 UTC m=+3541.409852333" Nov 02 23:47:51.950355 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:51.950319 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" Nov 02 23:47:54.328518 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:54.328464 5114 patch_prober.go:29] interesting pod/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 23:47:54.328978 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:54.328556 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" podUID="d0400368-a485-486f-b433-3c456aaddff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 23:47:54.991057 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:54.991022 5114 generic.go:356] "Generic (PLEG): container finished" podID="d0400368-a485-486f-b433-3c456aaddff9" containerID="7bb850e4a88cf684fae83c5159cbecd1e9b3c0c3a95d5407a969ea0156cc5623" exitCode=0 Nov 02 23:47:54.991289 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:54.991104 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" event={"ID":"d0400368-a485-486f-b433-3c456aaddff9","Type":"ContainerDied","Data":"7bb850e4a88cf684fae83c5159cbecd1e9b3c0c3a95d5407a969ea0156cc5623"} Nov 02 23:47:54.991627 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:54.991607 5114 scope.go:122] "RemoveContainer" containerID="7bb850e4a88cf684fae83c5159cbecd1e9b3c0c3a95d5407a969ea0156cc5623" Nov 02 23:47:54.993242 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:54.993215 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:47:54.993308 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:54.993289 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:47:54.993344 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:54.993331 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:47:55.999657 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:55.999620 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" event={"ID":"d0400368-a485-486f-b433-3c456aaddff9","Type":"ContainerStarted","Data":"6f8ab0398ebb42b754c3710d7442a4686550e44a5e2a443cf23ff114c5108ae9"} Nov 02 23:47:56.772927 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:56.772876 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:47:56.773228 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:56.773200 5114 patch_prober.go:29] interesting pod/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 23:47:56.773278 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:56.773255 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" podUID="d0400368-a485-486f-b433-3c456aaddff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 23:47:57.772682 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:57.772643 5114 patch_prober.go:29] interesting pod/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 23:47:57.773123 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:57.772704 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" podUID="d0400368-a485-486f-b433-3c456aaddff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 23:47:58.773336 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:58.773292 5114 patch_prober.go:29] interesting pod/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 23:47:58.773821 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:58.773360 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" podUID="d0400368-a485-486f-b433-3c456aaddff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 23:47:59.773107 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:59.773069 5114 patch_prober.go:29] interesting pod/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 23:47:59.773315 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:47:59.773141 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" podUID="d0400368-a485-486f-b433-3c456aaddff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 23:48:00.772279 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:00.772241 5114 patch_prober.go:29] interesting pod/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 23:48:00.772775 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:00.772301 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" podUID="d0400368-a485-486f-b433-3c456aaddff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 23:48:01.777646 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:01.777610 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:48:01.778091 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:01.777994 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:48:01.778705 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:01.778686 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:48:07.584359 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:48:07.584322 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:48:12.486699 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:12.486651 5114 patch_prober.go:29] interesting pod/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:48:12.487130 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:12.486730 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" podUID="6f16d749-1be4-4d77-bb08-21bcf16bb724" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:48:14.134801 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:14.134763 5114 generic.go:356] "Generic (PLEG): container finished" podID="6f16d749-1be4-4d77-bb08-21bcf16bb724" containerID="989320271dcb5483ab9e258b6f1ef2e26f3f2f69393dac19dbc9ffe21ec60ff4" exitCode=0 Nov 02 23:48:14.135250 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:14.134843 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" event={"ID":"6f16d749-1be4-4d77-bb08-21bcf16bb724","Type":"ContainerDied","Data":"989320271dcb5483ab9e258b6f1ef2e26f3f2f69393dac19dbc9ffe21ec60ff4"} Nov 02 23:48:14.135293 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:14.135258 5114 scope.go:122] "RemoveContainer" containerID="989320271dcb5483ab9e258b6f1ef2e26f3f2f69393dac19dbc9ffe21ec60ff4" Nov 02 23:48:14.136884 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:14.136858 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:48:14.136959 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:14.136927 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:48:14.136997 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:14.136973 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:48:15.143102 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:15.143062 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" event={"ID":"6f16d749-1be4-4d77-bb08-21bcf16bb724","Type":"ContainerStarted","Data":"a239b19ed4accc4184f3aab36f7e4b71577052c637e030f2303e5f14dd26a0d0"} Nov 02 23:48:15.584768 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:15.584727 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:48:15.585305 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:15.584917 5114 patch_prober.go:29] interesting pod/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 02 23:48:15.585305 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:15.584991 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" podUID="6f16d749-1be4-4d77-bb08-21bcf16bb724" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 02 23:48:16.585218 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:16.585181 5114 patch_prober.go:29] interesting pod/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 02 23:48:16.585681 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:16.585242 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" podUID="6f16d749-1be4-4d77-bb08-21bcf16bb724" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 02 23:48:17.585327 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:17.585288 5114 patch_prober.go:29] interesting pod/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 02 23:48:17.585884 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:17.585353 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" podUID="6f16d749-1be4-4d77-bb08-21bcf16bb724" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 02 23:48:18.584764 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:18.584727 5114 patch_prober.go:29] interesting pod/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 02 23:48:18.584972 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:18.584798 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" podUID="6f16d749-1be4-4d77-bb08-21bcf16bb724" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 02 23:48:19.584541 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:19.584467 5114 patch_prober.go:29] interesting pod/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 02 23:48:19.584999 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:19.584568 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" podUID="6f16d749-1be4-4d77-bb08-21bcf16bb724" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 02 23:48:20.588719 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:20.588686 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:48:20.589138 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:20.589103 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:48:20.589873 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:20.589852 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 02 23:48:27.046345 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:27.046303 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62647c61-bhpt4"] Nov 02 23:48:27.095670 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:27.095634 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62647c61-bhpt4"] Nov 02 23:48:28.794235 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:28.794198 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd51a61b-e410-4996-8e72-16bd6e6a7881" path="/var/lib/kubelet/pods/fd51a61b-e410-4996-8e72-16bd6e6a7881/volumes" Nov 02 23:48:32.371654 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.371611 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp"] Nov 02 23:48:32.375975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.375945 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.436664 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.436628 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-942137-openshift-default-dockercfg-7lg24\"" Nov 02 23:48:32.467899 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.467718 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wv7l\" (UniqueName: \"kubernetes.io/projected/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-kube-api-access-4wv7l\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.467899 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.467783 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-credential-socket\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.467899 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.467828 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-workload-certs\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.467899 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.467857 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-podinfo\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.467899 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.467895 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-envoy\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.468344 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.467956 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-data\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.468344 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.468022 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istiod-ca-cert\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.468344 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.468076 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-workload-socket\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.468344 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.468112 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-token\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.497025 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.496981 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp"] Nov 02 23:48:32.569201 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.569162 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wv7l\" (UniqueName: \"kubernetes.io/projected/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-kube-api-access-4wv7l\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.569201 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.569207 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-credential-socket\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.569514 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.569237 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-workload-certs\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.569514 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.569256 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-podinfo\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.569514 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.569275 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-envoy\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.569514 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.569311 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-data\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.569514 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.569360 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istiod-ca-cert\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.569514 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.569405 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-workload-socket\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.569514 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.569438 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-token\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.569941 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.569719 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-workload-certs\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.570283 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.570259 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istiod-ca-cert\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.570801 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.570776 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-data\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.570801 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.570790 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-credential-socket\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.570947 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.570894 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-workload-socket\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.573770 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.573741 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-envoy\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.574094 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.574069 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-podinfo\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.639255 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.639142 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-token\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.639425 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.639395 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wv7l\" (UniqueName: \"kubernetes.io/projected/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-kube-api-access-4wv7l\") pod \"e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:32.690459 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:32.690416 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:33.135575 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:33.135538 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp"] Nov 02 23:48:33.141842 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:33.141802 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:48:33.141975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:33.141879 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:48:33.141975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:33.141934 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:48:33.258753 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:33.257996 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" event={"ID":"5ca490fd-a101-4a30-9bce-ca42fbe5e92b","Type":"ContainerStarted","Data":"2c339568993d816a2375b9aef9a20dda76324a9adc0b7482fe0aad07098b2537"} Nov 02 23:48:33.258753 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:33.258067 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" event={"ID":"5ca490fd-a101-4a30-9bce-ca42fbe5e92b","Type":"ContainerStarted","Data":"a8eb95f7e7d8e9b02b8bb7d220e0147d470d12cd9ab039e6de7e5b47ae114247"} Nov 02 23:48:33.381689 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:33.381624 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" podStartSLOduration=1.3816035260000001 podStartE2EDuration="1.381603526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:48:33.37735013 +0000 UTC m=+3588.730383506" watchObservedRunningTime="2026-11-02 23:48:33.381603526 +0000 UTC m=+3588.734636901" Nov 02 23:48:33.691163 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:33.691102 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:34.697221 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:34.697190 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:34.813810 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:34.813771 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-875049-lbb59"] Nov 02 23:48:34.816650 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:34.816622 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-875049-lbb59"] Nov 02 23:48:34.997153 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:34.997059 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5"] Nov 02 23:48:34.997356 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:34.997300 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" podUID="fa21ce01-b481-4cd5-bbba-8e00e9f9769d" containerName="istio-proxy" containerID="cri-o://3db1cd58f0144e9c24ed5ac65cdfe0aa5511e0732e9a2fecf2564e31c6c2ee69" gracePeriod=30 Nov 02 23:48:35.270041 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:35.269951 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:35.271081 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:35.271061 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:48:36.794133 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:36.794100 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3764839-17c4-47af-b560-91d4bb3b06f1" path="/var/lib/kubelet/pods/f3764839-17c4-47af-b560-91d4bb3b06f1/volumes" Nov 02 23:48:37.584603 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:48:37.584561 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:48:38.045989 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:38.045952 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebe2f460-mlr4n"] Nov 02 23:48:38.095196 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:38.095147 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebe2f460-mlr4n"] Nov 02 23:48:38.793297 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:38.793261 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9705213-5a0d-4ea0-b8b9-98af78097933" path="/var/lib/kubelet/pods/b9705213-5a0d-4ea0-b8b9-98af78097933/volumes" Nov 02 23:48:40.200269 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.200240 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:48:40.276793 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.276712 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-workload-certs\") pod \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " Nov 02 23:48:40.276793 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.276749 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-credential-socket\") pod \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " Nov 02 23:48:40.276793 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.276782 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istiod-ca-cert\") pod \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " Nov 02 23:48:40.277071 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.276824 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-podinfo\") pod \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " Nov 02 23:48:40.277071 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.276910 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-envoy\") pod \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " Nov 02 23:48:40.277071 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.277025 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-workload-certs" pod "fa21ce01-b481-4cd5-bbba-8e00e9f9769d" (UID: "fa21ce01-b481-4cd5-bbba-8e00e9f9769d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:48:40.277071 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.277030 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-token\" (UniqueName: \"kubernetes.io/projected/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-token\") pod \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " Nov 02 23:48:40.277286 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.277091 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-credential-socket" pod "fa21ce01-b481-4cd5-bbba-8e00e9f9769d" (UID: "fa21ce01-b481-4cd5-bbba-8e00e9f9769d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:48:40.277286 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.277117 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-kube-api-access-7fkrf\" (UniqueName: \"kubernetes.io/projected/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-kube-api-access-7fkrf\") pod \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " Nov 02 23:48:40.277286 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.277144 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-data\") pod \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " Nov 02 23:48:40.277286 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.277170 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-workload-socket\") pod \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\" (UID: \"fa21ce01-b481-4cd5-bbba-8e00e9f9769d\") " Nov 02 23:48:40.277539 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.277521 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-data" pod "fa21ce01-b481-4cd5-bbba-8e00e9f9769d" (UID: "fa21ce01-b481-4cd5-bbba-8e00e9f9769d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:48:40.277725 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.277642 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:48:40.277725 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.277701 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:48:40.277725 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.277709 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-workload-socket" pod "fa21ce01-b481-4cd5-bbba-8e00e9f9769d" (UID: "fa21ce01-b481-4cd5-bbba-8e00e9f9769d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:48:40.277725 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.277712 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istiod-ca-cert" pod "fa21ce01-b481-4cd5-bbba-8e00e9f9769d" (UID: "fa21ce01-b481-4cd5-bbba-8e00e9f9769d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:48:40.277725 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.277719 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:48:40.279647 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.279622 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-envoy" pod "fa21ce01-b481-4cd5-bbba-8e00e9f9769d" (UID: "fa21ce01-b481-4cd5-bbba-8e00e9f9769d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:48:40.279647 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.279619 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-podinfo" pod "fa21ce01-b481-4cd5-bbba-8e00e9f9769d" (UID: "fa21ce01-b481-4cd5-bbba-8e00e9f9769d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:48:40.279866 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.279650 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-token" pod "fa21ce01-b481-4cd5-bbba-8e00e9f9769d" (UID: "fa21ce01-b481-4cd5-bbba-8e00e9f9769d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:48:40.280464 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.280447 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-kube-api-access-7fkrf" pod "fa21ce01-b481-4cd5-bbba-8e00e9f9769d" (UID: "fa21ce01-b481-4cd5-bbba-8e00e9f9769d"). InnerVolumeSpecName "kube-api-access-7fkrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:48:40.302712 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.302681 5114 generic.go:356] "Generic (PLEG): container finished" podID="fa21ce01-b481-4cd5-bbba-8e00e9f9769d" containerID="3db1cd58f0144e9c24ed5ac65cdfe0aa5511e0732e9a2fecf2564e31c6c2ee69" exitCode=0 Nov 02 23:48:40.302866 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.302748 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" Nov 02 23:48:40.302866 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.302753 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" event={"ID":"fa21ce01-b481-4cd5-bbba-8e00e9f9769d","Type":"ContainerDied","Data":"3db1cd58f0144e9c24ed5ac65cdfe0aa5511e0732e9a2fecf2564e31c6c2ee69"} Nov 02 23:48:40.302866 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.302806 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5" event={"ID":"fa21ce01-b481-4cd5-bbba-8e00e9f9769d","Type":"ContainerDied","Data":"25efc2fdcb753875f4891b29aaf0abb1bec7ca0555cf56bde6cfacbd9c72fd0b"} Nov 02 23:48:40.302866 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.302830 5114 scope.go:122] "RemoveContainer" containerID="3db1cd58f0144e9c24ed5ac65cdfe0aa5511e0732e9a2fecf2564e31c6c2ee69" Nov 02 23:48:40.314271 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.314249 5114 scope.go:122] "RemoveContainer" containerID="ddf7ba4f39f9b0ce871e7dc4047d4bcbaed49ae4eb612205e29296a9ce5deae0" Nov 02 23:48:40.324474 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.324454 5114 scope.go:122] "RemoveContainer" containerID="3db1cd58f0144e9c24ed5ac65cdfe0aa5511e0732e9a2fecf2564e31c6c2ee69" Nov 02 23:48:40.324771 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:48:40.324752 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3db1cd58f0144e9c24ed5ac65cdfe0aa5511e0732e9a2fecf2564e31c6c2ee69\": container with ID starting with 3db1cd58f0144e9c24ed5ac65cdfe0aa5511e0732e9a2fecf2564e31c6c2ee69 not found: ID does not exist" containerID="3db1cd58f0144e9c24ed5ac65cdfe0aa5511e0732e9a2fecf2564e31c6c2ee69" Nov 02 23:48:40.324844 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.324783 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db1cd58f0144e9c24ed5ac65cdfe0aa5511e0732e9a2fecf2564e31c6c2ee69"} err="failed to get container status \"3db1cd58f0144e9c24ed5ac65cdfe0aa5511e0732e9a2fecf2564e31c6c2ee69\": rpc error: code = NotFound desc = could not find container \"3db1cd58f0144e9c24ed5ac65cdfe0aa5511e0732e9a2fecf2564e31c6c2ee69\": container with ID starting with 3db1cd58f0144e9c24ed5ac65cdfe0aa5511e0732e9a2fecf2564e31c6c2ee69 not found: ID does not exist" Nov 02 23:48:40.324844 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.324801 5114 scope.go:122] "RemoveContainer" containerID="ddf7ba4f39f9b0ce871e7dc4047d4bcbaed49ae4eb612205e29296a9ce5deae0" Nov 02 23:48:40.325055 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:48:40.325038 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddf7ba4f39f9b0ce871e7dc4047d4bcbaed49ae4eb612205e29296a9ce5deae0\": container with ID starting with ddf7ba4f39f9b0ce871e7dc4047d4bcbaed49ae4eb612205e29296a9ce5deae0 not found: ID does not exist" containerID="ddf7ba4f39f9b0ce871e7dc4047d4bcbaed49ae4eb612205e29296a9ce5deae0" Nov 02 23:48:40.325104 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.325061 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddf7ba4f39f9b0ce871e7dc4047d4bcbaed49ae4eb612205e29296a9ce5deae0"} err="failed to get container status \"ddf7ba4f39f9b0ce871e7dc4047d4bcbaed49ae4eb612205e29296a9ce5deae0\": rpc error: code = NotFound desc = could not find container \"ddf7ba4f39f9b0ce871e7dc4047d4bcbaed49ae4eb612205e29296a9ce5deae0\": container with ID starting with ddf7ba4f39f9b0ce871e7dc4047d4bcbaed49ae4eb612205e29296a9ce5deae0 not found: ID does not exist" Nov 02 23:48:40.379616 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.379578 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:48:40.379616 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.379615 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fkrf\" (UniqueName: \"kubernetes.io/projected/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-kube-api-access-7fkrf\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:48:40.379825 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.379632 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:48:40.379825 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.379647 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:48:40.379825 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.379661 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:48:40.379825 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.379674 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa21ce01-b481-4cd5-bbba-8e00e9f9769d-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:48:40.481287 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.481252 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5"] Nov 02 23:48:40.485185 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.485157 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-875049-openshift-default-574965c6f-jzzx5"] Nov 02 23:48:40.793883 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:40.793845 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa21ce01-b481-4cd5-bbba-8e00e9f9769d" path="/var/lib/kubelet/pods/fa21ce01-b481-4cd5-bbba-8e00e9f9769d/volumes" Nov 02 23:48:45.605392 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.605351 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8"] Nov 02 23:48:45.605940 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.605844 5114 cpu_manager.go:414] "containerMap: removing container" podUID="fa21ce01-b481-4cd5-bbba-8e00e9f9769d" containerName="istio-proxy" Nov 02 23:48:45.605940 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.605858 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa21ce01-b481-4cd5-bbba-8e00e9f9769d" containerName="istio-proxy" Nov 02 23:48:45.606021 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.605974 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa21ce01-b481-4cd5-bbba-8e00e9f9769d" containerName="istio-proxy" Nov 02 23:48:45.606021 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.605983 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa21ce01-b481-4cd5-bbba-8e00e9f9769d" containerName="istio-proxy" Nov 02 23:48:45.610516 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.610496 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8" Nov 02 23:48:45.666031 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.665991 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8"] Nov 02 23:48:45.679008 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.678976 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-d75a69\"" Nov 02 23:48:45.712540 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.712504 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-d75a69-dockercfg-nq9d2\"" Nov 02 23:48:45.740813 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.740778 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/660a75de-19bf-4364-8a3c-bc2d04c64718-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8\" (UID: \"660a75de-19bf-4364-8a3c-bc2d04c64718\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8" Nov 02 23:48:45.741026 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.740839 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-782gk\" (UniqueName: \"kubernetes.io/projected/660a75de-19bf-4364-8a3c-bc2d04c64718-kube-api-access-782gk\") pod \"payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8\" (UID: \"660a75de-19bf-4364-8a3c-bc2d04c64718\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8" Nov 02 23:48:45.761636 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.761601 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg"] Nov 02 23:48:45.762077 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.762062 5114 cpu_manager.go:414] "containerMap: removing container" podUID="fa21ce01-b481-4cd5-bbba-8e00e9f9769d" containerName="istio-proxy" Nov 02 23:48:45.762139 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.762079 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa21ce01-b481-4cd5-bbba-8e00e9f9769d" containerName="istio-proxy" Nov 02 23:48:45.765393 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.765369 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg" Nov 02 23:48:45.824129 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.824096 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg"] Nov 02 23:48:45.842728 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.842688 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwndf\" (UniqueName: \"kubernetes.io/projected/5bd74059-d4b3-4a76-85c1-e3dbf32a25e7-kube-api-access-fwndf\") pod \"payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg\" (UID: \"5bd74059-d4b3-4a76-85c1-e3dbf32a25e7\") " pod="openshift-ingress/payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg" Nov 02 23:48:45.842982 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.842960 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/660a75de-19bf-4364-8a3c-bc2d04c64718-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8\" (UID: \"660a75de-19bf-4364-8a3c-bc2d04c64718\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8" Nov 02 23:48:45.843087 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.843070 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bd74059-d4b3-4a76-85c1-e3dbf32a25e7-plugins-config-volume\") pod \"payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg\" (UID: \"5bd74059-d4b3-4a76-85c1-e3dbf32a25e7\") " pod="openshift-ingress/payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg" Nov 02 23:48:45.843135 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.843109 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-782gk\" (UniqueName: \"kubernetes.io/projected/660a75de-19bf-4364-8a3c-bc2d04c64718-kube-api-access-782gk\") pod \"payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8\" (UID: \"660a75de-19bf-4364-8a3c-bc2d04c64718\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8" Nov 02 23:48:45.843729 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.843711 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/660a75de-19bf-4364-8a3c-bc2d04c64718-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8\" (UID: \"660a75de-19bf-4364-8a3c-bc2d04c64718\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8" Nov 02 23:48:45.904084 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.904053 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-782gk\" (UniqueName: \"kubernetes.io/projected/660a75de-19bf-4364-8a3c-bc2d04c64718-kube-api-access-782gk\") pod \"payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8\" (UID: \"660a75de-19bf-4364-8a3c-bc2d04c64718\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8" Nov 02 23:48:45.924647 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.924613 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8" Nov 02 23:48:45.944149 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.944107 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bd74059-d4b3-4a76-85c1-e3dbf32a25e7-plugins-config-volume\") pod \"payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg\" (UID: \"5bd74059-d4b3-4a76-85c1-e3dbf32a25e7\") " pod="openshift-ingress/payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg" Nov 02 23:48:45.944399 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.944222 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwndf\" (UniqueName: \"kubernetes.io/projected/5bd74059-d4b3-4a76-85c1-e3dbf32a25e7-kube-api-access-fwndf\") pod \"payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg\" (UID: \"5bd74059-d4b3-4a76-85c1-e3dbf32a25e7\") " pod="openshift-ingress/payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg" Nov 02 23:48:45.945246 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:45.945227 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bd74059-d4b3-4a76-85c1-e3dbf32a25e7-plugins-config-volume\") pod \"payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg\" (UID: \"5bd74059-d4b3-4a76-85c1-e3dbf32a25e7\") " pod="openshift-ingress/payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg" Nov 02 23:48:46.006525 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:46.005875 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwndf\" (UniqueName: \"kubernetes.io/projected/5bd74059-d4b3-4a76-85c1-e3dbf32a25e7-kube-api-access-fwndf\") pod \"payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg\" (UID: \"5bd74059-d4b3-4a76-85c1-e3dbf32a25e7\") " pod="openshift-ingress/payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg" Nov 02 23:48:46.079251 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:46.079201 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg" Nov 02 23:48:46.301861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:46.301831 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8"] Nov 02 23:48:46.302896 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:48:46.302869 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660a75de_19bf_4364_8a3c_bc2d04c64718.slice/crio-5556f81202d1d6608cceeedd44fef7c7e189e0c253c6078467ea52d327579394 WatchSource:0}: Error finding container 5556f81202d1d6608cceeedd44fef7c7e189e0c253c6078467ea52d327579394: Status 404 returned error can't find the container with id 5556f81202d1d6608cceeedd44fef7c7e189e0c253c6078467ea52d327579394 Nov 02 23:48:46.340589 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:46.340551 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8" event={"ID":"660a75de-19bf-4364-8a3c-bc2d04c64718","Type":"ContainerStarted","Data":"5556f81202d1d6608cceeedd44fef7c7e189e0c253c6078467ea52d327579394"} Nov 02 23:48:46.460040 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:46.460010 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg"] Nov 02 23:48:46.461676 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:48:46.461644 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd74059_d4b3_4a76_85c1_e3dbf32a25e7.slice/crio-a6be9098b0edde74b37865440cab601275b2ed02a3ff52571e7f7f6789642781 WatchSource:0}: Error finding container a6be9098b0edde74b37865440cab601275b2ed02a3ff52571e7f7f6789642781: Status 404 returned error can't find the container with id a6be9098b0edde74b37865440cab601275b2ed02a3ff52571e7f7f6789642781 Nov 02 23:48:47.349165 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:47.349119 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8" event={"ID":"660a75de-19bf-4364-8a3c-bc2d04c64718","Type":"ContainerStarted","Data":"1b05b859af907fab8cf5e98b35d370ddfb2ff7ba8f3c300b476a08eb58287c1c"} Nov 02 23:48:47.349697 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:47.349194 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8" Nov 02 23:48:47.350849 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:47.350819 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg" event={"ID":"5bd74059-d4b3-4a76-85c1-e3dbf32a25e7","Type":"ContainerStarted","Data":"2d73fcf533ca7f2f3127c0aa8e0ad114f0feb113de9dae95c86a215d736d975d"} Nov 02 23:48:47.350972 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:47.350865 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg" event={"ID":"5bd74059-d4b3-4a76-85c1-e3dbf32a25e7","Type":"ContainerStarted","Data":"a6be9098b0edde74b37865440cab601275b2ed02a3ff52571e7f7f6789642781"} Nov 02 23:48:47.350972 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:47.350888 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg" Nov 02 23:48:47.463714 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:47.463660 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8" podStartSLOduration=2.463645188 podStartE2EDuration="2.463645188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:48:47.463090053 +0000 UTC m=+3602.816123429" watchObservedRunningTime="2026-11-02 23:48:47.463645188 +0000 UTC m=+3602.816678563" Nov 02 23:48:47.580012 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:47.579956 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg" podStartSLOduration=2.579937402 podStartE2EDuration="2.579937402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:48:47.577646537 +0000 UTC m=+3602.930679912" watchObservedRunningTime="2026-11-02 23:48:47.579937402 +0000 UTC m=+3602.932970783" Nov 02 23:48:58.359683 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:58.359646 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg" Nov 02 23:48:58.360119 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:48:58.359750 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8" Nov 02 23:49:07.584829 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:49:07.584789 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:49:08.363559 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:08.363122 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69"] Nov 02 23:49:08.380003 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:08.369038 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69" Nov 02 23:49:08.384867 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:08.384601 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb7px\" (UniqueName: \"kubernetes.io/projected/69bf017e-78f8-46c9-95eb-08647f5d1788-kube-api-access-sb7px\") pod \"payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69\" (UID: \"69bf017e-78f8-46c9-95eb-08647f5d1788\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69" Nov 02 23:49:08.384867 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:08.384701 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69bf017e-78f8-46c9-95eb-08647f5d1788-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69\" (UID: \"69bf017e-78f8-46c9-95eb-08647f5d1788\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69" Nov 02 23:49:08.421985 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:08.421768 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-056584-dockercfg-n4tfz\"" Nov 02 23:49:08.422184 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:08.422047 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-056584\"" Nov 02 23:49:08.424794 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:08.424762 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69"] Nov 02 23:49:08.485853 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:08.485797 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69bf017e-78f8-46c9-95eb-08647f5d1788-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69\" (UID: \"69bf017e-78f8-46c9-95eb-08647f5d1788\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69" Nov 02 23:49:08.486059 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:08.485960 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb7px\" (UniqueName: \"kubernetes.io/projected/69bf017e-78f8-46c9-95eb-08647f5d1788-kube-api-access-sb7px\") pod \"payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69\" (UID: \"69bf017e-78f8-46c9-95eb-08647f5d1788\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69" Nov 02 23:49:08.486675 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:08.486656 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69bf017e-78f8-46c9-95eb-08647f5d1788-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69\" (UID: \"69bf017e-78f8-46c9-95eb-08647f5d1788\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69" Nov 02 23:49:08.514770 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:08.514736 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v"] Nov 02 23:49:08.518602 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:08.518581 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v" Nov 02 23:49:08.546834 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:08.546793 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb7px\" (UniqueName: \"kubernetes.io/projected/69bf017e-78f8-46c9-95eb-08647f5d1788-kube-api-access-sb7px\") pod \"payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69\" (UID: \"69bf017e-78f8-46c9-95eb-08647f5d1788\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69" Nov 02 23:49:08.576887 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:08.576853 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v"] Nov 02 23:49:08.586733 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:08.586696 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w42t9\" (UniqueName: \"kubernetes.io/projected/2775490f-e402-4fa8-af5a-df684e8a0198-kube-api-access-w42t9\") pod \"payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v\" (UID: \"2775490f-e402-4fa8-af5a-df684e8a0198\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v" Nov 02 23:49:08.587179 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:08.586826 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2775490f-e402-4fa8-af5a-df684e8a0198-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v\" (UID: \"2775490f-e402-4fa8-af5a-df684e8a0198\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v" Nov 02 23:49:08.688722 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:08.688612 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w42t9\" (UniqueName: \"kubernetes.io/projected/2775490f-e402-4fa8-af5a-df684e8a0198-kube-api-access-w42t9\") pod \"payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v\" (UID: \"2775490f-e402-4fa8-af5a-df684e8a0198\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v" Nov 02 23:49:08.688722 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:08.688688 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2775490f-e402-4fa8-af5a-df684e8a0198-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v\" (UID: \"2775490f-e402-4fa8-af5a-df684e8a0198\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v" Nov 02 23:49:08.689436 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:08.689415 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2775490f-e402-4fa8-af5a-df684e8a0198-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v\" (UID: \"2775490f-e402-4fa8-af5a-df684e8a0198\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v" Nov 02 23:49:08.692471 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:08.692449 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69" Nov 02 23:49:08.750106 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:08.750070 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w42t9\" (UniqueName: \"kubernetes.io/projected/2775490f-e402-4fa8-af5a-df684e8a0198-kube-api-access-w42t9\") pod \"payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v\" (UID: \"2775490f-e402-4fa8-af5a-df684e8a0198\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v" Nov 02 23:49:08.832227 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:08.832197 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v" Nov 02 23:49:09.068557 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:09.068519 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69"] Nov 02 23:49:09.070400 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:49:09.070363 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69bf017e_78f8_46c9_95eb_08647f5d1788.slice/crio-aab6e4f08119b318a47214f3750166f91210dc5ef59b7d121f259653b13b337a WatchSource:0}: Error finding container aab6e4f08119b318a47214f3750166f91210dc5ef59b7d121f259653b13b337a: Status 404 returned error can't find the container with id aab6e4f08119b318a47214f3750166f91210dc5ef59b7d121f259653b13b337a Nov 02 23:49:09.212833 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:09.212804 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v"] Nov 02 23:49:09.214814 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:49:09.214782 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2775490f_e402_4fa8_af5a_df684e8a0198.slice/crio-4353549857b58872aec49287b81bff2c32b7b8d1f7f7974bf3d7a8e76aea1243 WatchSource:0}: Error finding container 4353549857b58872aec49287b81bff2c32b7b8d1f7f7974bf3d7a8e76aea1243: Status 404 returned error can't find the container with id 4353549857b58872aec49287b81bff2c32b7b8d1f7f7974bf3d7a8e76aea1243 Nov 02 23:49:09.487698 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:09.487597 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69" event={"ID":"69bf017e-78f8-46c9-95eb-08647f5d1788","Type":"ContainerStarted","Data":"93ced4f4c566b8fe24e1aa9944c300f2e6306629a8df91d1a49b22b7a8301653"} Nov 02 23:49:09.487698 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:09.487658 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69" event={"ID":"69bf017e-78f8-46c9-95eb-08647f5d1788","Type":"ContainerStarted","Data":"aab6e4f08119b318a47214f3750166f91210dc5ef59b7d121f259653b13b337a"} Nov 02 23:49:09.487698 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:09.487686 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69" Nov 02 23:49:09.489419 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:09.489386 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v" event={"ID":"2775490f-e402-4fa8-af5a-df684e8a0198","Type":"ContainerStarted","Data":"9fd3f2ff0dc721a9c01bbe5c01e6d798eae9e3b6b169064c5a909ddcc646fa77"} Nov 02 23:49:09.489586 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:09.489438 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v" event={"ID":"2775490f-e402-4fa8-af5a-df684e8a0198","Type":"ContainerStarted","Data":"4353549857b58872aec49287b81bff2c32b7b8d1f7f7974bf3d7a8e76aea1243"} Nov 02 23:49:09.489586 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:09.489491 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v" Nov 02 23:49:09.604553 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:09.604499 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69" podStartSLOduration=1.604468888 podStartE2EDuration="1.604468888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:49:09.603161678 +0000 UTC m=+3624.956195054" watchObservedRunningTime="2026-11-02 23:49:09.604468888 +0000 UTC m=+3624.957502264" Nov 02 23:49:09.719135 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:09.719074 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v" podStartSLOduration=1.719059359 podStartE2EDuration="1.719059359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:49:09.718405108 +0000 UTC m=+3625.071438496" watchObservedRunningTime="2026-11-02 23:49:09.719059359 +0000 UTC m=+3625.072092734" Nov 02 23:49:15.047065 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:15.047022 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f4298b7-sqdqg"] Nov 02 23:49:15.095995 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:15.095953 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f4298b7-sqdqg"] Nov 02 23:49:16.794920 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:16.794888 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0330f53-39ae-458a-8a80-e80734e2c768" path="/var/lib/kubelet/pods/e0330f53-39ae-458a-8a80-e80734e2c768/volumes" Nov 02 23:49:20.499161 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:20.499132 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v" Nov 02 23:49:20.499639 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:20.499207 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69" Nov 02 23:49:24.861226 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:24.861166 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl"] Nov 02 23:49:24.865377 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:24.865347 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:49:24.969673 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:24.969636 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-882a2a\"/\"openshift-service-ca.crt\"" Nov 02 23:49:24.969956 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:24.969926 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-882a2a\"/\"auth-test-model-882a2a-kserve-self-signed-certs\"" Nov 02 23:49:24.970059 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:24.969964 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-882a2a\"/\"kube-root-ca.crt\"" Nov 02 23:49:24.970059 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:24.970043 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-882a2a\"/\"default-dockercfg-cj25g\"" Nov 02 23:49:24.972472 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:24.972451 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl"] Nov 02 23:49:24.978659 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:24.978638 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-tmp-dir\") pod \"auth-test-model-882a2a-kserve-64f9bf596f-lbhtl\" (UID: \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\") " pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:49:24.978747 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:24.978693 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-dshm\") pod \"auth-test-model-882a2a-kserve-64f9bf596f-lbhtl\" (UID: \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\") " pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:49:24.978838 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:24.978817 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-model-cache\") pod \"auth-test-model-882a2a-kserve-64f9bf596f-lbhtl\" (UID: \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\") " pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:49:24.978903 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:24.978887 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npvjb\" (UniqueName: \"kubernetes.io/projected/642fc4ab-f938-46bb-9ee1-eb815c9d956b-kube-api-access-npvjb\") pod \"auth-test-model-882a2a-kserve-64f9bf596f-lbhtl\" (UID: \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\") " pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:49:24.979011 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:24.978995 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/642fc4ab-f938-46bb-9ee1-eb815c9d956b-tls-certs\") pod \"auth-test-model-882a2a-kserve-64f9bf596f-lbhtl\" (UID: \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\") " pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:49:24.979080 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:24.979057 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-home\") pod \"auth-test-model-882a2a-kserve-64f9bf596f-lbhtl\" (UID: \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\") " pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:49:25.080793 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:25.080754 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npvjb\" (UniqueName: \"kubernetes.io/projected/642fc4ab-f938-46bb-9ee1-eb815c9d956b-kube-api-access-npvjb\") pod \"auth-test-model-882a2a-kserve-64f9bf596f-lbhtl\" (UID: \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\") " pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:49:25.081020 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:25.080856 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/642fc4ab-f938-46bb-9ee1-eb815c9d956b-tls-certs\") pod \"auth-test-model-882a2a-kserve-64f9bf596f-lbhtl\" (UID: \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\") " pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:49:25.081020 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:25.080901 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-home\") pod \"auth-test-model-882a2a-kserve-64f9bf596f-lbhtl\" (UID: \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\") " pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:49:25.081020 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:25.080980 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-tmp-dir\") pod \"auth-test-model-882a2a-kserve-64f9bf596f-lbhtl\" (UID: \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\") " pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:49:25.081207 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:25.081047 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-dshm\") pod \"auth-test-model-882a2a-kserve-64f9bf596f-lbhtl\" (UID: \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\") " pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:49:25.081207 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:25.081101 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-model-cache\") pod \"auth-test-model-882a2a-kserve-64f9bf596f-lbhtl\" (UID: \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\") " pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:49:25.081582 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:25.081550 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-home\") pod \"auth-test-model-882a2a-kserve-64f9bf596f-lbhtl\" (UID: \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\") " pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:49:25.081696 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:25.081586 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-model-cache\") pod \"auth-test-model-882a2a-kserve-64f9bf596f-lbhtl\" (UID: \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\") " pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:49:25.081696 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:25.081595 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-tmp-dir\") pod \"auth-test-model-882a2a-kserve-64f9bf596f-lbhtl\" (UID: \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\") " pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:49:25.083559 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:25.083534 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-dshm\") pod \"auth-test-model-882a2a-kserve-64f9bf596f-lbhtl\" (UID: \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\") " pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:49:25.083742 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:25.083725 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/642fc4ab-f938-46bb-9ee1-eb815c9d956b-tls-certs\") pod \"auth-test-model-882a2a-kserve-64f9bf596f-lbhtl\" (UID: \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\") " pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:49:25.141937 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:25.141837 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npvjb\" (UniqueName: \"kubernetes.io/projected/642fc4ab-f938-46bb-9ee1-eb815c9d956b-kube-api-access-npvjb\") pod \"auth-test-model-882a2a-kserve-64f9bf596f-lbhtl\" (UID: \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\") " pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:49:25.179400 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:25.179358 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:49:25.566964 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:25.566932 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl"] Nov 02 23:49:25.568388 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:49:25.568358 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod642fc4ab_f938_46bb_9ee1_eb815c9d956b.slice/crio-f07962043935f24c6a05a7113ee4921073aaf5fb01ce84ba66178698ef0c7016 WatchSource:0}: Error finding container f07962043935f24c6a05a7113ee4921073aaf5fb01ce84ba66178698ef0c7016: Status 404 returned error can't find the container with id f07962043935f24c6a05a7113ee4921073aaf5fb01ce84ba66178698ef0c7016 Nov 02 23:49:25.609368 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:25.609322 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" event={"ID":"642fc4ab-f938-46bb-9ee1-eb815c9d956b","Type":"ContainerStarted","Data":"f07962043935f24c6a05a7113ee4921073aaf5fb01ce84ba66178698ef0c7016"} Nov 02 23:49:26.055767 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:26.055727 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f30cfd3a-m7tmd"] Nov 02 23:49:26.058965 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:26.058941 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f30cfd3a-m7tmd"] Nov 02 23:49:26.616556 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:26.616520 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" event={"ID":"642fc4ab-f938-46bb-9ee1-eb815c9d956b","Type":"ContainerStarted","Data":"d922f16cd3a1d18b95f15850d673a9b49cf9aca3e717b9d19d3e979a0a94cba7"} Nov 02 23:49:26.733278 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:26.733215 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" podStartSLOduration=2.733199571 podStartE2EDuration="2.733199571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:49:26.731831136 +0000 UTC m=+3642.084864514" watchObservedRunningTime="2026-11-02 23:49:26.733199571 +0000 UTC m=+3642.086232946" Nov 02 23:49:26.793996 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:26.793957 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00962ce7-ca9b-450b-b68d-bbb0165a7553" path="/var/lib/kubelet/pods/00962ce7-ca9b-450b-b68d-bbb0165a7553/volumes" Nov 02 23:49:27.180515 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:27.180430 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:49:35.180520 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:35.180468 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:49:35.193323 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:35.193296 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:49:35.682435 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:35.682395 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:49:37.583976 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:49:37.583929 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:49:44.050806 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:44.050767 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f141ce6-dqg6b"] Nov 02 23:49:44.099361 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:44.099320 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f141ce6-dqg6b"] Nov 02 23:49:44.794802 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:44.794766 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bea96b8-5ae1-425c-98de-c90603572fbc" path="/var/lib/kubelet/pods/1bea96b8-5ae1-425c-98de-c90603572fbc/volumes" Nov 02 23:49:45.053553 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:45.053434 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc"] Nov 02 23:49:45.865460 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:45.865426 5114 scope.go:122] "RemoveContainer" containerID="8bf11aedd8636603f471383fe35af645c8022c13a6d1d7a5377100d1ad21e8e9" Nov 02 23:49:45.874885 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:45.874861 5114 scope.go:122] "RemoveContainer" containerID="a2c8210681df33031f5bc9a35c4abbc43c98b71f7c1ad76e9affd4974604cd5d" Nov 02 23:49:50.054719 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:50.054648 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc" podUID="d286a90d-42ec-4b33-8180-1a6afcc1b3ee" containerName="payload-pre-processing" containerID="cri-o://2d70292124389839d322e4f632fe3ba2d72f9b607152d80cbd6d2e9d0a4860d4" gracePeriod=25 Nov 02 23:49:50.333302 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:50.333271 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc" Nov 02 23:49:50.469104 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:50.469065 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d286a90d-42ec-4b33-8180-1a6afcc1b3ee-kube-api-access-w56pq\" (UniqueName: \"kubernetes.io/projected/d286a90d-42ec-4b33-8180-1a6afcc1b3ee-kube-api-access-w56pq\") pod \"d286a90d-42ec-4b33-8180-1a6afcc1b3ee\" (UID: \"d286a90d-42ec-4b33-8180-1a6afcc1b3ee\") " Nov 02 23:49:50.469281 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:50.469231 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d286a90d-42ec-4b33-8180-1a6afcc1b3ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d286a90d-42ec-4b33-8180-1a6afcc1b3ee-plugins-config-volume\") pod \"d286a90d-42ec-4b33-8180-1a6afcc1b3ee\" (UID: \"d286a90d-42ec-4b33-8180-1a6afcc1b3ee\") " Nov 02 23:49:50.469684 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:50.469655 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d286a90d-42ec-4b33-8180-1a6afcc1b3ee-plugins-config-volume" pod "d286a90d-42ec-4b33-8180-1a6afcc1b3ee" (UID: "d286a90d-42ec-4b33-8180-1a6afcc1b3ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:49:50.471412 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:50.471385 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d286a90d-42ec-4b33-8180-1a6afcc1b3ee-kube-api-access-w56pq" pod "d286a90d-42ec-4b33-8180-1a6afcc1b3ee" (UID: "d286a90d-42ec-4b33-8180-1a6afcc1b3ee"). InnerVolumeSpecName "kube-api-access-w56pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:49:50.570539 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:50.570425 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w56pq\" (UniqueName: \"kubernetes.io/projected/d286a90d-42ec-4b33-8180-1a6afcc1b3ee-kube-api-access-w56pq\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:49:50.570539 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:50.570459 5114 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d286a90d-42ec-4b33-8180-1a6afcc1b3ee-plugins-config-volume\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:49:50.763304 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:50.763261 5114 generic.go:356] "Generic (PLEG): container finished" podID="d286a90d-42ec-4b33-8180-1a6afcc1b3ee" containerID="2d70292124389839d322e4f632fe3ba2d72f9b607152d80cbd6d2e9d0a4860d4" exitCode=0 Nov 02 23:49:50.763501 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:50.763313 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc" Nov 02 23:49:50.763501 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:50.763346 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc" event={"ID":"d286a90d-42ec-4b33-8180-1a6afcc1b3ee","Type":"ContainerDied","Data":"2d70292124389839d322e4f632fe3ba2d72f9b607152d80cbd6d2e9d0a4860d4"} Nov 02 23:49:50.763501 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:50.763406 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc" event={"ID":"d286a90d-42ec-4b33-8180-1a6afcc1b3ee","Type":"ContainerDied","Data":"6153aa3a135ab22907b9bbda06553a425f614f45ba3bea48cdf04c3d57292064"} Nov 02 23:49:50.763501 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:50.763434 5114 scope.go:122] "RemoveContainer" containerID="2d70292124389839d322e4f632fe3ba2d72f9b607152d80cbd6d2e9d0a4860d4" Nov 02 23:49:50.774110 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:50.774071 5114 scope.go:122] "RemoveContainer" containerID="2d70292124389839d322e4f632fe3ba2d72f9b607152d80cbd6d2e9d0a4860d4" Nov 02 23:49:50.774405 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:49:50.774382 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d70292124389839d322e4f632fe3ba2d72f9b607152d80cbd6d2e9d0a4860d4\": container with ID starting with 2d70292124389839d322e4f632fe3ba2d72f9b607152d80cbd6d2e9d0a4860d4 not found: ID does not exist" containerID="2d70292124389839d322e4f632fe3ba2d72f9b607152d80cbd6d2e9d0a4860d4" Nov 02 23:49:50.774518 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:50.774417 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d70292124389839d322e4f632fe3ba2d72f9b607152d80cbd6d2e9d0a4860d4"} err="failed to get container status \"2d70292124389839d322e4f632fe3ba2d72f9b607152d80cbd6d2e9d0a4860d4\": rpc error: code = NotFound desc = could not find container \"2d70292124389839d322e4f632fe3ba2d72f9b607152d80cbd6d2e9d0a4860d4\": container with ID starting with 2d70292124389839d322e4f632fe3ba2d72f9b607152d80cbd6d2e9d0a4860d4 not found: ID does not exist" Nov 02 23:49:50.927283 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:50.927195 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc"] Nov 02 23:49:50.931136 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:50.931106 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f49ab-54d488c79f-j9rwc"] Nov 02 23:49:52.794127 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:52.794085 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d286a90d-42ec-4b33-8180-1a6afcc1b3ee" path="/var/lib/kubelet/pods/d286a90d-42ec-4b33-8180-1a6afcc1b3ee/volumes" Nov 02 23:49:55.048372 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:55.048314 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2d54581-266r9"] Nov 02 23:49:55.097841 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:55.097804 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2d54581-266r9"] Nov 02 23:49:56.794006 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:49:56.793974 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a54093d-f5a1-41af-b5db-deb571d294ed" path="/var/lib/kubelet/pods/1a54093d-f5a1-41af-b5db-deb571d294ed/volumes" Nov 02 23:50:07.584853 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:50:07.584809 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:50:32.021668 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:32.021562 5114 generic.go:356] "Generic (PLEG): container finished" podID="5ca490fd-a101-4a30-9bce-ca42fbe5e92b" containerID="2c339568993d816a2375b9aef9a20dda76324a9adc0b7482fe0aad07098b2537" exitCode=0 Nov 02 23:50:32.021668 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:32.021647 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" event={"ID":"5ca490fd-a101-4a30-9bce-ca42fbe5e92b","Type":"ContainerDied","Data":"2c339568993d816a2375b9aef9a20dda76324a9adc0b7482fe0aad07098b2537"} Nov 02 23:50:32.022309 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:32.022193 5114 scope.go:122] "RemoveContainer" containerID="2c339568993d816a2375b9aef9a20dda76324a9adc0b7482fe0aad07098b2537" Nov 02 23:50:32.024052 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:32.024018 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:50:32.024163 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:32.024103 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:50:32.024163 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:32.024136 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:50:33.048960 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:33.048916 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" event={"ID":"5ca490fd-a101-4a30-9bce-ca42fbe5e92b","Type":"ContainerStarted","Data":"3b1c27791388cbfa892ab8598025648924b67bdddd74f2336f8c38f339afc6a6"} Nov 02 23:50:33.691171 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:33.691132 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:50:33.692628 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:33.692601 5114 patch_prober.go:29] interesting pod/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 02 23:50:33.692761 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:33.692658 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" podUID="5ca490fd-a101-4a30-9bce-ca42fbe5e92b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 02 23:50:34.691812 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:34.691769 5114 patch_prober.go:29] interesting pod/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 02 23:50:34.692250 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:34.691828 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" podUID="5ca490fd-a101-4a30-9bce-ca42fbe5e92b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 02 23:50:35.270244 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:35.270187 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:50:35.690957 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:35.690906 5114 patch_prober.go:29] interesting pod/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 02 23:50:35.691165 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:35.690974 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" podUID="5ca490fd-a101-4a30-9bce-ca42fbe5e92b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 02 23:50:36.691816 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:36.691774 5114 patch_prober.go:29] interesting pod/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 02 23:50:36.692261 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:36.691829 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" podUID="5ca490fd-a101-4a30-9bce-ca42fbe5e92b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 02 23:50:37.584842 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:50:37.584799 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:50:37.691738 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:37.691701 5114 patch_prober.go:29] interesting pod/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 02 23:50:37.691925 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:37.691770 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" podUID="5ca490fd-a101-4a30-9bce-ca42fbe5e92b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 02 23:50:38.696314 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:38.696282 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:50:38.697329 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:38.697306 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:50:39.061775 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.061741 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4"] Nov 02 23:50:39.062299 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.062282 5114 cpu_manager.go:414] "containerMap: removing container" podUID="d286a90d-42ec-4b33-8180-1a6afcc1b3ee" containerName="payload-pre-processing" Nov 02 23:50:39.062368 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.062301 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d286a90d-42ec-4b33-8180-1a6afcc1b3ee" containerName="payload-pre-processing" Nov 02 23:50:39.062406 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.062394 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="d286a90d-42ec-4b33-8180-1a6afcc1b3ee" containerName="payload-pre-processing" Nov 02 23:50:39.065684 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.065668 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:50:39.117807 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.117777 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-6c3d8e\"/\"openshift-service-ca.crt\"" Nov 02 23:50:39.121855 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.121824 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-6c3d8e\"/\"auth-test-model-6c3d8e-kserve-self-signed-certs\"" Nov 02 23:50:39.122039 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.121895 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-6c3d8e\"/\"default-dockercfg-96758\"" Nov 02 23:50:39.124522 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.124471 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4"] Nov 02 23:50:39.190109 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.190069 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-tmp-dir\") pod \"auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4\" (UID: \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\") " pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:50:39.190109 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.190106 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-model-cache\") pod \"auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4\" (UID: \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\") " pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:50:39.190350 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.190132 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-home\") pod \"auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4\" (UID: \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\") " pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:50:39.190404 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.190386 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgl27\" (UniqueName: \"kubernetes.io/projected/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-kube-api-access-hgl27\") pod \"auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4\" (UID: \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\") " pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:50:39.190446 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.190433 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-tls-certs\") pod \"auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4\" (UID: \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\") " pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:50:39.190635 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.190607 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-dshm\") pod \"auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4\" (UID: \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\") " pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:50:39.218472 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.218441 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-6c3d8e\"/\"kube-root-ca.crt\"" Nov 02 23:50:39.291849 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.291812 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgl27\" (UniqueName: \"kubernetes.io/projected/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-kube-api-access-hgl27\") pod \"auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4\" (UID: \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\") " pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:50:39.291849 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.291852 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-tls-certs\") pod \"auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4\" (UID: \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\") " pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:50:39.292153 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.291968 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-dshm\") pod \"auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4\" (UID: \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\") " pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:50:39.292153 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.292002 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-tmp-dir\") pod \"auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4\" (UID: \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\") " pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:50:39.292153 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.292020 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-model-cache\") pod \"auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4\" (UID: \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\") " pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:50:39.292153 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.292037 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-home\") pod \"auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4\" (UID: \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\") " pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:50:39.292503 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.292454 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-home\") pod \"auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4\" (UID: \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\") " pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:50:39.292637 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.292565 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-tmp-dir\") pod \"auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4\" (UID: \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\") " pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:50:39.292768 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.292747 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-model-cache\") pod \"auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4\" (UID: \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\") " pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:50:39.295413 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.295376 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-tls-certs\") pod \"auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4\" (UID: \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\") " pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:50:39.295584 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.295442 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-dshm\") pod \"auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4\" (UID: \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\") " pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:50:39.353392 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.353304 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgl27\" (UniqueName: \"kubernetes.io/projected/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-kube-api-access-hgl27\") pod \"auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4\" (UID: \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\") " pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:50:39.377866 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.377822 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:50:39.766587 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:39.766559 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4"] Nov 02 23:50:39.768900 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:50:39.768865 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podacd46b0c_b1e7_465e_a871_bef07e0f5e4e.slice/crio-14a37d04433c264ba16a38d2b76f3fb9cee173c942bf6002809f3cbe8f4e80d1 WatchSource:0}: Error finding container 14a37d04433c264ba16a38d2b76f3fb9cee173c942bf6002809f3cbe8f4e80d1: Status 404 returned error can't find the container with id 14a37d04433c264ba16a38d2b76f3fb9cee173c942bf6002809f3cbe8f4e80d1 Nov 02 23:50:40.104676 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:40.104641 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" event={"ID":"acd46b0c-b1e7-465e-a871-bef07e0f5e4e","Type":"ContainerStarted","Data":"4d7926b1e13759743ba7e6da554ec0d68a0d71e03b2dc6c000cf233453f51a19"} Nov 02 23:50:40.104676 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:40.104685 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" event={"ID":"acd46b0c-b1e7-465e-a871-bef07e0f5e4e","Type":"ContainerStarted","Data":"14a37d04433c264ba16a38d2b76f3fb9cee173c942bf6002809f3cbe8f4e80d1"} Nov 02 23:50:40.221870 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:40.221811 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" podStartSLOduration=1.221796241 podStartE2EDuration="1.221796241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:50:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:50:40.21978733 +0000 UTC m=+3715.572820729" watchObservedRunningTime="2026-11-02 23:50:40.221796241 +0000 UTC m=+3715.574829616" Nov 02 23:50:40.378598 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:40.378514 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:50:45.904597 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:45.904566 5114 scope.go:122] "RemoveContainer" containerID="3522547d8af74e06b25cbbe9b908758e1c59616933c8d0f6b9f82f2af3b385ad" Nov 02 23:50:45.914852 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:45.914831 5114 scope.go:122] "RemoveContainer" containerID="19d3f6afbc1bb0c3b3aba2a4f0fa6e3b9f2aec194448f5fc5f62321711713280" Nov 02 23:50:45.924381 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:45.924360 5114 scope.go:122] "RemoveContainer" containerID="bdac3c3fd7c7bc7d821d524aeba070eb623fd4f88ca3c67c347e99e6551fb4dd" Nov 02 23:50:49.378013 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:49.377976 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:50:49.391557 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:49.391526 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:50:50.177092 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:50:50.177057 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:51:07.584786 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:51:07.584744 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:51:09.084170 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:09.084127 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g"] Nov 02 23:51:09.088010 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:09.087986 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g" Nov 02 23:51:09.129900 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:09.129861 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708cbf69-febc-4f09-a2d6-518e77977bf7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g\" (UID: \"708cbf69-febc-4f09-a2d6-518e77977bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g" Nov 02 23:51:09.130088 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:09.130045 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkfhx\" (UniqueName: \"kubernetes.io/projected/708cbf69-febc-4f09-a2d6-518e77977bf7-kube-api-access-rkfhx\") pod \"maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g\" (UID: \"708cbf69-febc-4f09-a2d6-518e77977bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g" Nov 02 23:51:09.130164 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:09.130149 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708cbf69-febc-4f09-a2d6-518e77977bf7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g\" (UID: \"708cbf69-febc-4f09-a2d6-518e77977bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g" Nov 02 23:51:09.142376 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:09.142345 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:51:09.191574 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:09.191540 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:51:09.193994 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:09.193972 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g"] Nov 02 23:51:09.231235 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:09.231196 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708cbf69-febc-4f09-a2d6-518e77977bf7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g\" (UID: \"708cbf69-febc-4f09-a2d6-518e77977bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g" Nov 02 23:51:09.231461 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:09.231261 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708cbf69-febc-4f09-a2d6-518e77977bf7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g\" (UID: \"708cbf69-febc-4f09-a2d6-518e77977bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g" Nov 02 23:51:09.231461 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:09.231358 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkfhx\" (UniqueName: \"kubernetes.io/projected/708cbf69-febc-4f09-a2d6-518e77977bf7-kube-api-access-rkfhx\") pod \"maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g\" (UID: \"708cbf69-febc-4f09-a2d6-518e77977bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g" Nov 02 23:51:09.231984 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:09.231964 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708cbf69-febc-4f09-a2d6-518e77977bf7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g\" (UID: \"708cbf69-febc-4f09-a2d6-518e77977bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g" Nov 02 23:51:09.233817 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:09.233791 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708cbf69-febc-4f09-a2d6-518e77977bf7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g\" (UID: \"708cbf69-febc-4f09-a2d6-518e77977bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g" Nov 02 23:51:09.292224 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:09.292180 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkfhx\" (UniqueName: \"kubernetes.io/projected/708cbf69-febc-4f09-a2d6-518e77977bf7-kube-api-access-rkfhx\") pod \"maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g\" (UID: \"708cbf69-febc-4f09-a2d6-518e77977bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g" Nov 02 23:51:09.402398 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:09.402290 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g" Nov 02 23:51:09.779950 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:09.779918 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g"] Nov 02 23:51:09.781627 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:51:09.781596 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708cbf69_febc_4f09_a2d6_518e77977bf7.slice/crio-aa5f4577a343ed57f778eab9e6e69d5c0c04b7b396869d4b4076096b967a2242 WatchSource:0}: Error finding container aa5f4577a343ed57f778eab9e6e69d5c0c04b7b396869d4b4076096b967a2242: Status 404 returned error can't find the container with id aa5f4577a343ed57f778eab9e6e69d5c0c04b7b396869d4b4076096b967a2242 Nov 02 23:51:10.288388 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:10.288292 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g" event={"ID":"708cbf69-febc-4f09-a2d6-518e77977bf7","Type":"ContainerStarted","Data":"a71a4a5156e524e48122c5870bed2c7b3ddd11f91be914e3dd7e37902f2008dd"} Nov 02 23:51:10.288388 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:10.288355 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g" event={"ID":"708cbf69-febc-4f09-a2d6-518e77977bf7","Type":"ContainerStarted","Data":"aa5f4577a343ed57f778eab9e6e69d5c0c04b7b396869d4b4076096b967a2242"} Nov 02 23:51:10.402801 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:10.402740 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g" podStartSLOduration=1.402724132 podStartE2EDuration="1.402724132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:51:10.402371564 +0000 UTC m=+3745.755404941" watchObservedRunningTime="2026-11-02 23:51:10.402724132 +0000 UTC m=+3745.755757519" Nov 02 23:51:11.300717 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:11.300678 5114 generic.go:356] "Generic (PLEG): container finished" podID="708cbf69-febc-4f09-a2d6-518e77977bf7" containerID="a71a4a5156e524e48122c5870bed2c7b3ddd11f91be914e3dd7e37902f2008dd" exitCode=0 Nov 02 23:51:11.301197 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:11.300769 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g" event={"ID":"708cbf69-febc-4f09-a2d6-518e77977bf7","Type":"ContainerDied","Data":"a71a4a5156e524e48122c5870bed2c7b3ddd11f91be914e3dd7e37902f2008dd"} Nov 02 23:51:12.468904 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:12.468877 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g" Nov 02 23:51:12.566337 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:12.566242 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/708cbf69-febc-4f09-a2d6-518e77977bf7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708cbf69-febc-4f09-a2d6-518e77977bf7-maas-api-openshift-ca\") pod \"708cbf69-febc-4f09-a2d6-518e77977bf7\" (UID: \"708cbf69-febc-4f09-a2d6-518e77977bf7\") " Nov 02 23:51:12.566541 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:12.566338 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708cbf69-febc-4f09-a2d6-518e77977bf7-kube-api-access-rkfhx\" (UniqueName: \"kubernetes.io/projected/708cbf69-febc-4f09-a2d6-518e77977bf7-kube-api-access-rkfhx\") pod \"708cbf69-febc-4f09-a2d6-518e77977bf7\" (UID: \"708cbf69-febc-4f09-a2d6-518e77977bf7\") " Nov 02 23:51:12.566541 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:12.566529 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/708cbf69-febc-4f09-a2d6-518e77977bf7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708cbf69-febc-4f09-a2d6-518e77977bf7-maas-api-xks-ca\") pod \"708cbf69-febc-4f09-a2d6-518e77977bf7\" (UID: \"708cbf69-febc-4f09-a2d6-518e77977bf7\") " Nov 02 23:51:12.566754 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:12.566728 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708cbf69-febc-4f09-a2d6-518e77977bf7-maas-api-openshift-ca" pod "708cbf69-febc-4f09-a2d6-518e77977bf7" (UID: "708cbf69-febc-4f09-a2d6-518e77977bf7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:51:12.567012 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:12.566989 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708cbf69-febc-4f09-a2d6-518e77977bf7-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:51:12.568815 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:12.568793 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708cbf69-febc-4f09-a2d6-518e77977bf7-kube-api-access-rkfhx" pod "708cbf69-febc-4f09-a2d6-518e77977bf7" (UID: "708cbf69-febc-4f09-a2d6-518e77977bf7"). InnerVolumeSpecName "kube-api-access-rkfhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:51:12.568815 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:12.568796 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708cbf69-febc-4f09-a2d6-518e77977bf7-maas-api-xks-ca" pod "708cbf69-febc-4f09-a2d6-518e77977bf7" (UID: "708cbf69-febc-4f09-a2d6-518e77977bf7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:51:12.668423 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:12.668374 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708cbf69-febc-4f09-a2d6-518e77977bf7-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:51:12.668423 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:12.668415 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkfhx\" (UniqueName: \"kubernetes.io/projected/708cbf69-febc-4f09-a2d6-518e77977bf7-kube-api-access-rkfhx\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:51:13.314566 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:13.314519 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g" event={"ID":"708cbf69-febc-4f09-a2d6-518e77977bf7","Type":"ContainerDied","Data":"aa5f4577a343ed57f778eab9e6e69d5c0c04b7b396869d4b4076096b967a2242"} Nov 02 23:51:13.314747 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:13.314583 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa5f4577a343ed57f778eab9e6e69d5c0c04b7b396869d4b4076096b967a2242" Nov 02 23:51:13.314747 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:13.314531 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g" Nov 02 23:51:37.584865 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:51:37.584825 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:51:45.964213 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:45.964181 5114 scope.go:122] "RemoveContainer" containerID="03fa83d65492bc6a39bb75e81e07a93cadb574c4bf08bde6db3d7418f41edc05" Nov 02 23:51:47.258695 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:47.258658 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz"] Nov 02 23:51:47.369744 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:47.369699 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj"] Nov 02 23:51:48.184957 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:48.184904 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg"] Nov 02 23:51:48.238113 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:48.238075 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8"] Nov 02 23:51:52.259918 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.259866 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz" podUID="084ed54a-9cca-405e-a481-6933da8821dc" containerName="payload-processing" containerID="cri-o://f44671821d5cddfe15685c9a8ab0016ca910360025b58ace5ad6a1ec0992ed68" gracePeriod=25 Nov 02 23:51:52.370202 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.370135 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj" podUID="7773e0dc-6fee-4e97-9db3-9106268ac5c4" containerName="payload-pre-processing" containerID="cri-o://c151504f67df962367f2b889c810765d6a1b2be6d9ef7732e7611d6fcb70cd41" gracePeriod=25 Nov 02 23:51:52.536962 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.536917 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz" Nov 02 23:51:52.547738 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.547698 5114 generic.go:356] "Generic (PLEG): container finished" podID="084ed54a-9cca-405e-a481-6933da8821dc" containerID="f44671821d5cddfe15685c9a8ab0016ca910360025b58ace5ad6a1ec0992ed68" exitCode=0 Nov 02 23:51:52.547953 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.547767 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz" Nov 02 23:51:52.547953 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.547773 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz" event={"ID":"084ed54a-9cca-405e-a481-6933da8821dc","Type":"ContainerDied","Data":"f44671821d5cddfe15685c9a8ab0016ca910360025b58ace5ad6a1ec0992ed68"} Nov 02 23:51:52.547953 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.547831 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz" event={"ID":"084ed54a-9cca-405e-a481-6933da8821dc","Type":"ContainerDied","Data":"7af11e96eb74fecfb9f3d5027d98fbbb78e1e5edcfb46b4605c2ed5b791e18d6"} Nov 02 23:51:52.547953 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.547863 5114 scope.go:122] "RemoveContainer" containerID="f44671821d5cddfe15685c9a8ab0016ca910360025b58ace5ad6a1ec0992ed68" Nov 02 23:51:52.550410 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.550381 5114 generic.go:356] "Generic (PLEG): container finished" podID="7773e0dc-6fee-4e97-9db3-9106268ac5c4" containerID="c151504f67df962367f2b889c810765d6a1b2be6d9ef7732e7611d6fcb70cd41" exitCode=0 Nov 02 23:51:52.550555 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.550439 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj" event={"ID":"7773e0dc-6fee-4e97-9db3-9106268ac5c4","Type":"ContainerDied","Data":"c151504f67df962367f2b889c810765d6a1b2be6d9ef7732e7611d6fcb70cd41"} Nov 02 23:51:52.560689 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.560575 5114 scope.go:122] "RemoveContainer" containerID="f44671821d5cddfe15685c9a8ab0016ca910360025b58ace5ad6a1ec0992ed68" Nov 02 23:51:52.561009 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:51:52.560985 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f44671821d5cddfe15685c9a8ab0016ca910360025b58ace5ad6a1ec0992ed68\": container with ID starting with f44671821d5cddfe15685c9a8ab0016ca910360025b58ace5ad6a1ec0992ed68 not found: ID does not exist" containerID="f44671821d5cddfe15685c9a8ab0016ca910360025b58ace5ad6a1ec0992ed68" Nov 02 23:51:52.561106 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.561020 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f44671821d5cddfe15685c9a8ab0016ca910360025b58ace5ad6a1ec0992ed68"} err="failed to get container status \"f44671821d5cddfe15685c9a8ab0016ca910360025b58ace5ad6a1ec0992ed68\": rpc error: code = NotFound desc = could not find container \"f44671821d5cddfe15685c9a8ab0016ca910360025b58ace5ad6a1ec0992ed68\": container with ID starting with f44671821d5cddfe15685c9a8ab0016ca910360025b58ace5ad6a1ec0992ed68 not found: ID does not exist" Nov 02 23:51:52.637967 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.637929 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/084ed54a-9cca-405e-a481-6933da8821dc-kube-api-access-c84mw\" (UniqueName: \"kubernetes.io/projected/084ed54a-9cca-405e-a481-6933da8821dc-kube-api-access-c84mw\") pod \"084ed54a-9cca-405e-a481-6933da8821dc\" (UID: \"084ed54a-9cca-405e-a481-6933da8821dc\") " Nov 02 23:51:52.638133 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.638046 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/084ed54a-9cca-405e-a481-6933da8821dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/084ed54a-9cca-405e-a481-6933da8821dc-plugins-config-volume\") pod \"084ed54a-9cca-405e-a481-6933da8821dc\" (UID: \"084ed54a-9cca-405e-a481-6933da8821dc\") " Nov 02 23:51:52.638513 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.638467 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/084ed54a-9cca-405e-a481-6933da8821dc-plugins-config-volume" pod "084ed54a-9cca-405e-a481-6933da8821dc" (UID: "084ed54a-9cca-405e-a481-6933da8821dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:51:52.640616 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.640591 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084ed54a-9cca-405e-a481-6933da8821dc-kube-api-access-c84mw" pod "084ed54a-9cca-405e-a481-6933da8821dc" (UID: "084ed54a-9cca-405e-a481-6933da8821dc"). InnerVolumeSpecName "kube-api-access-c84mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:51:52.649602 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.649570 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj" Nov 02 23:51:52.739936 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.739897 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7773e0dc-6fee-4e97-9db3-9106268ac5c4-kube-api-access-nhvz2\" (UniqueName: \"kubernetes.io/projected/7773e0dc-6fee-4e97-9db3-9106268ac5c4-kube-api-access-nhvz2\") pod \"7773e0dc-6fee-4e97-9db3-9106268ac5c4\" (UID: \"7773e0dc-6fee-4e97-9db3-9106268ac5c4\") " Nov 02 23:51:52.740143 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.740048 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7773e0dc-6fee-4e97-9db3-9106268ac5c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7773e0dc-6fee-4e97-9db3-9106268ac5c4-plugins-config-volume\") pod \"7773e0dc-6fee-4e97-9db3-9106268ac5c4\" (UID: \"7773e0dc-6fee-4e97-9db3-9106268ac5c4\") " Nov 02 23:51:52.740308 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.740292 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c84mw\" (UniqueName: \"kubernetes.io/projected/084ed54a-9cca-405e-a481-6933da8821dc-kube-api-access-c84mw\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:51:52.740382 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.740307 5114 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/084ed54a-9cca-405e-a481-6933da8821dc-plugins-config-volume\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:51:52.740437 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.740412 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7773e0dc-6fee-4e97-9db3-9106268ac5c4-plugins-config-volume" pod "7773e0dc-6fee-4e97-9db3-9106268ac5c4" (UID: "7773e0dc-6fee-4e97-9db3-9106268ac5c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:51:52.742356 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.742327 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7773e0dc-6fee-4e97-9db3-9106268ac5c4-kube-api-access-nhvz2" pod "7773e0dc-6fee-4e97-9db3-9106268ac5c4" (UID: "7773e0dc-6fee-4e97-9db3-9106268ac5c4"). InnerVolumeSpecName "kube-api-access-nhvz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:51:52.841558 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.841517 5114 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7773e0dc-6fee-4e97-9db3-9106268ac5c4-plugins-config-volume\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:51:52.841558 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.841561 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhvz2\" (UniqueName: \"kubernetes.io/projected/7773e0dc-6fee-4e97-9db3-9106268ac5c4-kube-api-access-nhvz2\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:51:52.963256 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.963220 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz"] Nov 02 23:51:52.967320 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:52.967288 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3058ee-7cc58f4cb7-cqnsz"] Nov 02 23:51:53.185582 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.185453 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg" podUID="5bd74059-d4b3-4a76-85c1-e3dbf32a25e7" containerName="payload-processing" containerID="cri-o://2d73fcf533ca7f2f3127c0aa8e0ad114f0feb113de9dae95c86a215d736d975d" gracePeriod=25 Nov 02 23:51:53.238780 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.238728 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8" podUID="660a75de-19bf-4364-8a3c-bc2d04c64718" containerName="payload-pre-processing" containerID="cri-o://1b05b859af907fab8cf5e98b35d370ddfb2ff7ba8f3c300b476a08eb58287c1c" gracePeriod=25 Nov 02 23:51:53.509842 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.509813 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg" Nov 02 23:51:53.547340 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.547308 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8" Nov 02 23:51:53.556956 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.556918 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj" Nov 02 23:51:53.557151 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.556954 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj" event={"ID":"7773e0dc-6fee-4e97-9db3-9106268ac5c4","Type":"ContainerDied","Data":"431df346754247a47622d33d418a7c98ace2df5002053a4529398eaf15a0c4ee"} Nov 02 23:51:53.557151 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.557033 5114 scope.go:122] "RemoveContainer" containerID="c151504f67df962367f2b889c810765d6a1b2be6d9ef7732e7611d6fcb70cd41" Nov 02 23:51:53.566263 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.561067 5114 generic.go:356] "Generic (PLEG): container finished" podID="660a75de-19bf-4364-8a3c-bc2d04c64718" containerID="1b05b859af907fab8cf5e98b35d370ddfb2ff7ba8f3c300b476a08eb58287c1c" exitCode=0 Nov 02 23:51:53.566263 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.561264 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8" Nov 02 23:51:53.566263 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.561261 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8" event={"ID":"660a75de-19bf-4364-8a3c-bc2d04c64718","Type":"ContainerDied","Data":"1b05b859af907fab8cf5e98b35d370ddfb2ff7ba8f3c300b476a08eb58287c1c"} Nov 02 23:51:53.566263 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.561572 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8" event={"ID":"660a75de-19bf-4364-8a3c-bc2d04c64718","Type":"ContainerDied","Data":"5556f81202d1d6608cceeedd44fef7c7e189e0c253c6078467ea52d327579394"} Nov 02 23:51:53.566263 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.564321 5114 generic.go:356] "Generic (PLEG): container finished" podID="5bd74059-d4b3-4a76-85c1-e3dbf32a25e7" containerID="2d73fcf533ca7f2f3127c0aa8e0ad114f0feb113de9dae95c86a215d736d975d" exitCode=0 Nov 02 23:51:53.566263 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.564455 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg" event={"ID":"5bd74059-d4b3-4a76-85c1-e3dbf32a25e7","Type":"ContainerDied","Data":"2d73fcf533ca7f2f3127c0aa8e0ad114f0feb113de9dae95c86a215d736d975d"} Nov 02 23:51:53.566263 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.564532 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg" event={"ID":"5bd74059-d4b3-4a76-85c1-e3dbf32a25e7","Type":"ContainerDied","Data":"a6be9098b0edde74b37865440cab601275b2ed02a3ff52571e7f7f6789642781"} Nov 02 23:51:53.566263 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.564646 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg" Nov 02 23:51:53.574355 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.574327 5114 scope.go:122] "RemoveContainer" containerID="1b05b859af907fab8cf5e98b35d370ddfb2ff7ba8f3c300b476a08eb58287c1c" Nov 02 23:51:53.584777 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.584755 5114 scope.go:122] "RemoveContainer" containerID="1b05b859af907fab8cf5e98b35d370ddfb2ff7ba8f3c300b476a08eb58287c1c" Nov 02 23:51:53.585143 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:51:53.585120 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b05b859af907fab8cf5e98b35d370ddfb2ff7ba8f3c300b476a08eb58287c1c\": container with ID starting with 1b05b859af907fab8cf5e98b35d370ddfb2ff7ba8f3c300b476a08eb58287c1c not found: ID does not exist" containerID="1b05b859af907fab8cf5e98b35d370ddfb2ff7ba8f3c300b476a08eb58287c1c" Nov 02 23:51:53.585219 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.585158 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b05b859af907fab8cf5e98b35d370ddfb2ff7ba8f3c300b476a08eb58287c1c"} err="failed to get container status \"1b05b859af907fab8cf5e98b35d370ddfb2ff7ba8f3c300b476a08eb58287c1c\": rpc error: code = NotFound desc = could not find container \"1b05b859af907fab8cf5e98b35d370ddfb2ff7ba8f3c300b476a08eb58287c1c\": container with ID starting with 1b05b859af907fab8cf5e98b35d370ddfb2ff7ba8f3c300b476a08eb58287c1c not found: ID does not exist" Nov 02 23:51:53.585219 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.585187 5114 scope.go:122] "RemoveContainer" containerID="2d73fcf533ca7f2f3127c0aa8e0ad114f0feb113de9dae95c86a215d736d975d" Nov 02 23:51:53.594684 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.594664 5114 scope.go:122] "RemoveContainer" containerID="2d73fcf533ca7f2f3127c0aa8e0ad114f0feb113de9dae95c86a215d736d975d" Nov 02 23:51:53.595000 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:51:53.594974 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d73fcf533ca7f2f3127c0aa8e0ad114f0feb113de9dae95c86a215d736d975d\": container with ID starting with 2d73fcf533ca7f2f3127c0aa8e0ad114f0feb113de9dae95c86a215d736d975d not found: ID does not exist" containerID="2d73fcf533ca7f2f3127c0aa8e0ad114f0feb113de9dae95c86a215d736d975d" Nov 02 23:51:53.595067 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.595014 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d73fcf533ca7f2f3127c0aa8e0ad114f0feb113de9dae95c86a215d736d975d"} err="failed to get container status \"2d73fcf533ca7f2f3127c0aa8e0ad114f0feb113de9dae95c86a215d736d975d\": rpc error: code = NotFound desc = could not find container \"2d73fcf533ca7f2f3127c0aa8e0ad114f0feb113de9dae95c86a215d736d975d\": container with ID starting with 2d73fcf533ca7f2f3127c0aa8e0ad114f0feb113de9dae95c86a215d736d975d not found: ID does not exist" Nov 02 23:51:53.650402 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.650365 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bd74059-d4b3-4a76-85c1-e3dbf32a25e7-kube-api-access-fwndf\" (UniqueName: \"kubernetes.io/projected/5bd74059-d4b3-4a76-85c1-e3dbf32a25e7-kube-api-access-fwndf\") pod \"5bd74059-d4b3-4a76-85c1-e3dbf32a25e7\" (UID: \"5bd74059-d4b3-4a76-85c1-e3dbf32a25e7\") " Nov 02 23:51:53.650615 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.650448 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/660a75de-19bf-4364-8a3c-bc2d04c64718-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/660a75de-19bf-4364-8a3c-bc2d04c64718-plugins-config-volume\") pod \"660a75de-19bf-4364-8a3c-bc2d04c64718\" (UID: \"660a75de-19bf-4364-8a3c-bc2d04c64718\") " Nov 02 23:51:53.650671 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.650648 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660a75de-19bf-4364-8a3c-bc2d04c64718-kube-api-access-782gk\" (UniqueName: \"kubernetes.io/projected/660a75de-19bf-4364-8a3c-bc2d04c64718-kube-api-access-782gk\") pod \"660a75de-19bf-4364-8a3c-bc2d04c64718\" (UID: \"660a75de-19bf-4364-8a3c-bc2d04c64718\") " Nov 02 23:51:53.650720 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.650682 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bd74059-d4b3-4a76-85c1-e3dbf32a25e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bd74059-d4b3-4a76-85c1-e3dbf32a25e7-plugins-config-volume\") pod \"5bd74059-d4b3-4a76-85c1-e3dbf32a25e7\" (UID: \"5bd74059-d4b3-4a76-85c1-e3dbf32a25e7\") " Nov 02 23:51:53.650950 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.650915 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660a75de-19bf-4364-8a3c-bc2d04c64718-plugins-config-volume" pod "660a75de-19bf-4364-8a3c-bc2d04c64718" (UID: "660a75de-19bf-4364-8a3c-bc2d04c64718"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:51:53.651215 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.651187 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bd74059-d4b3-4a76-85c1-e3dbf32a25e7-plugins-config-volume" pod "5bd74059-d4b3-4a76-85c1-e3dbf32a25e7" (UID: "5bd74059-d4b3-4a76-85c1-e3dbf32a25e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:51:53.651372 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.651349 5114 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/660a75de-19bf-4364-8a3c-bc2d04c64718-plugins-config-volume\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:51:53.651416 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.651380 5114 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bd74059-d4b3-4a76-85c1-e3dbf32a25e7-plugins-config-volume\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:51:53.653400 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.653379 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd74059-d4b3-4a76-85c1-e3dbf32a25e7-kube-api-access-fwndf" pod "5bd74059-d4b3-4a76-85c1-e3dbf32a25e7" (UID: "5bd74059-d4b3-4a76-85c1-e3dbf32a25e7"). InnerVolumeSpecName "kube-api-access-fwndf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:51:53.653465 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.653410 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660a75de-19bf-4364-8a3c-bc2d04c64718-kube-api-access-782gk" pod "660a75de-19bf-4364-8a3c-bc2d04c64718" (UID: "660a75de-19bf-4364-8a3c-bc2d04c64718"). InnerVolumeSpecName "kube-api-access-782gk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:51:53.752770 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.752665 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwndf\" (UniqueName: \"kubernetes.io/projected/5bd74059-d4b3-4a76-85c1-e3dbf32a25e7-kube-api-access-fwndf\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:51:53.752770 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.752709 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-782gk\" (UniqueName: \"kubernetes.io/projected/660a75de-19bf-4364-8a3c-bc2d04c64718-kube-api-access-782gk\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:51:53.788107 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.788060 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj"] Nov 02 23:51:53.791829 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:53.791799 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3058ee-7fcbc566f5-6zfxj"] Nov 02 23:51:54.015323 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:54.015233 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg"] Nov 02 23:51:54.021210 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:54.021172 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-d75a69-79bfd87f49-2kvmg"] Nov 02 23:51:54.127902 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:54.127860 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8"] Nov 02 23:51:54.131997 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:54.131959 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-d75a69-7c4b8f9975-hk9h8"] Nov 02 23:51:54.793583 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:54.793550 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="084ed54a-9cca-405e-a481-6933da8821dc" path="/var/lib/kubelet/pods/084ed54a-9cca-405e-a481-6933da8821dc/volumes" Nov 02 23:51:54.794009 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:54.793993 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bd74059-d4b3-4a76-85c1-e3dbf32a25e7" path="/var/lib/kubelet/pods/5bd74059-d4b3-4a76-85c1-e3dbf32a25e7/volumes" Nov 02 23:51:54.794371 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:54.794359 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660a75de-19bf-4364-8a3c-bc2d04c64718" path="/var/lib/kubelet/pods/660a75de-19bf-4364-8a3c-bc2d04c64718/volumes" Nov 02 23:51:54.794758 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:51:54.794742 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7773e0dc-6fee-4e97-9db3-9106268ac5c4" path="/var/lib/kubelet/pods/7773e0dc-6fee-4e97-9db3-9106268ac5c4/volumes" Nov 02 23:52:02.133279 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.133237 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf"] Nov 02 23:52:02.133732 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.133714 5114 cpu_manager.go:414] "containerMap: removing container" podUID="660a75de-19bf-4364-8a3c-bc2d04c64718" containerName="payload-pre-processing" Nov 02 23:52:02.133786 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.133731 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660a75de-19bf-4364-8a3c-bc2d04c64718" containerName="payload-pre-processing" Nov 02 23:52:02.133786 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.133752 5114 cpu_manager.go:414] "containerMap: removing container" podUID="708cbf69-febc-4f09-a2d6-518e77977bf7" containerName="revoke-keys" Nov 02 23:52:02.133786 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.133759 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708cbf69-febc-4f09-a2d6-518e77977bf7" containerName="revoke-keys" Nov 02 23:52:02.133897 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.133790 5114 cpu_manager.go:414] "containerMap: removing container" podUID="084ed54a-9cca-405e-a481-6933da8821dc" containerName="payload-processing" Nov 02 23:52:02.133897 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.133801 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084ed54a-9cca-405e-a481-6933da8821dc" containerName="payload-processing" Nov 02 23:52:02.133897 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.133827 5114 cpu_manager.go:414] "containerMap: removing container" podUID="7773e0dc-6fee-4e97-9db3-9106268ac5c4" containerName="payload-pre-processing" Nov 02 23:52:02.133897 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.133835 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7773e0dc-6fee-4e97-9db3-9106268ac5c4" containerName="payload-pre-processing" Nov 02 23:52:02.133897 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.133862 5114 cpu_manager.go:414] "containerMap: removing container" podUID="5bd74059-d4b3-4a76-85c1-e3dbf32a25e7" containerName="payload-processing" Nov 02 23:52:02.133897 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.133867 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bd74059-d4b3-4a76-85c1-e3dbf32a25e7" containerName="payload-processing" Nov 02 23:52:02.134099 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.133975 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="708cbf69-febc-4f09-a2d6-518e77977bf7" containerName="revoke-keys" Nov 02 23:52:02.134099 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.133985 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="084ed54a-9cca-405e-a481-6933da8821dc" containerName="payload-processing" Nov 02 23:52:02.134099 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.133992 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="7773e0dc-6fee-4e97-9db3-9106268ac5c4" containerName="payload-pre-processing" Nov 02 23:52:02.134099 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.134000 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bd74059-d4b3-4a76-85c1-e3dbf32a25e7" containerName="payload-processing" Nov 02 23:52:02.134099 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.134008 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="660a75de-19bf-4364-8a3c-bc2d04c64718" containerName="payload-pre-processing" Nov 02 23:52:02.139051 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.139025 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.166220 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.166172 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-istio-envoy\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.167420 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.167386 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrnjf\" (UniqueName: \"kubernetes.io/projected/9ab21771-e42c-462f-b83f-ee47948059dd-kube-api-access-lrnjf\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.167746 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.167723 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ab21771-e42c-462f-b83f-ee47948059dd-istiod-ca-cert\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.167929 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.167912 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-workload-certs\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.168122 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.168107 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-credential-socket\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.168314 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.168297 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-istio-data\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.168456 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.168441 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ab21771-e42c-462f-b83f-ee47948059dd-istio-podinfo\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.168644 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.168626 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ab21771-e42c-462f-b83f-ee47948059dd-istio-token\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.168789 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.168774 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-workload-socket\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.197937 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.196765 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-0f7002-openshift-default-dockercfg-tcnvv\"" Nov 02 23:52:02.213767 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.212457 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf"] Nov 02 23:52:02.277141 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.277094 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ab21771-e42c-462f-b83f-ee47948059dd-istiod-ca-cert\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.277346 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.277168 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-workload-certs\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.277346 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.277231 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-credential-socket\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.277346 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.277314 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-istio-data\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.277346 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.277341 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ab21771-e42c-462f-b83f-ee47948059dd-istio-podinfo\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.277605 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.277378 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ab21771-e42c-462f-b83f-ee47948059dd-istio-token\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.277605 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.277416 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-workload-socket\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.277605 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.277537 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-istio-envoy\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.277772 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.277633 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrnjf\" (UniqueName: \"kubernetes.io/projected/9ab21771-e42c-462f-b83f-ee47948059dd-kube-api-access-lrnjf\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.277956 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.277926 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-istio-data\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.278025 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.278006 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-workload-certs\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.278105 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.278081 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ab21771-e42c-462f-b83f-ee47948059dd-istiod-ca-cert\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.278627 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.278599 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-credential-socket\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.278912 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.278865 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-workload-socket\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.286930 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.286881 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-istio-envoy\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.287646 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.287623 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ab21771-e42c-462f-b83f-ee47948059dd-istio-podinfo\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.338852 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.338808 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ab21771-e42c-462f-b83f-ee47948059dd-istio-token\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.339040 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.338940 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrnjf\" (UniqueName: \"kubernetes.io/projected/9ab21771-e42c-462f-b83f-ee47948059dd-kube-api-access-lrnjf\") pod \"e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.452546 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.452415 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:02.886622 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.886593 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf"] Nov 02 23:52:02.889177 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:52:02.889141 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab21771_e42c_462f_b83f_ee47948059dd.slice/crio-25c7af79c95ff3a227a8532e18e01600dfc5201671720f4ca3a6aeaedf8e9331 WatchSource:0}: Error finding container 25c7af79c95ff3a227a8532e18e01600dfc5201671720f4ca3a6aeaedf8e9331: Status 404 returned error can't find the container with id 25c7af79c95ff3a227a8532e18e01600dfc5201671720f4ca3a6aeaedf8e9331 Nov 02 23:52:02.892041 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.891999 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:52:02.892172 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.892078 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:52:02.892172 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:02.892135 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:52:03.629539 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:03.629498 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" event={"ID":"9ab21771-e42c-462f-b83f-ee47948059dd","Type":"ContainerStarted","Data":"defe1bd2fff79a70b520b761a6f75def4df0e187ed8b7b9b339e8c1d1e1bba1e"} Nov 02 23:52:03.630036 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:03.629566 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" event={"ID":"9ab21771-e42c-462f-b83f-ee47948059dd","Type":"ContainerStarted","Data":"25c7af79c95ff3a227a8532e18e01600dfc5201671720f4ca3a6aeaedf8e9331"} Nov 02 23:52:03.750089 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:03.750030 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" podStartSLOduration=1.750015546 podStartE2EDuration="1.750015546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:52:03.748459622 +0000 UTC m=+3799.101493021" watchObservedRunningTime="2026-11-02 23:52:03.750015546 +0000 UTC m=+3799.103048920" Nov 02 23:52:04.452788 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:04.452740 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:04.458414 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:04.458384 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:04.636539 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:04.636509 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:04.637740 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:04.637714 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:52:07.586987 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:52:07.586946 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:52:13.686704 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:13.686668 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg"] Nov 02 23:52:13.795858 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:13.795822 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq"] Nov 02 23:52:18.687799 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:18.687708 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg" podUID="b5551d34-16b0-4ed8-97e1-0d49e3d01096" containerName="payload-processing" containerID="cri-o://5821b30b00386914ff3b1dc64b9fb4c75b6fca2e7a21b841ea35d4fb3738403f" gracePeriod=25 Nov 02 23:52:18.796349 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:18.796309 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq" podUID="b82e0d2a-6469-4f21-966a-a2048217f811" containerName="payload-pre-processing" containerID="cri-o://f4c31ccedbc2ec044cb766737e85e285c5b8c5173036a4512f37f489e6490654" gracePeriod=25 Nov 02 23:52:18.956428 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:18.956343 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg" Nov 02 23:52:19.072356 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.072324 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq" Nov 02 23:52:19.077356 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.077324 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5551d34-16b0-4ed8-97e1-0d49e3d01096-kube-api-access-ppcvl\" (UniqueName: \"kubernetes.io/projected/b5551d34-16b0-4ed8-97e1-0d49e3d01096-kube-api-access-ppcvl\") pod \"b5551d34-16b0-4ed8-97e1-0d49e3d01096\" (UID: \"b5551d34-16b0-4ed8-97e1-0d49e3d01096\") " Nov 02 23:52:19.077554 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.077397 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5551d34-16b0-4ed8-97e1-0d49e3d01096-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5551d34-16b0-4ed8-97e1-0d49e3d01096-plugins-config-volume\") pod \"b5551d34-16b0-4ed8-97e1-0d49e3d01096\" (UID: \"b5551d34-16b0-4ed8-97e1-0d49e3d01096\") " Nov 02 23:52:19.078034 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.077838 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5551d34-16b0-4ed8-97e1-0d49e3d01096-plugins-config-volume" pod "b5551d34-16b0-4ed8-97e1-0d49e3d01096" (UID: "b5551d34-16b0-4ed8-97e1-0d49e3d01096"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:52:19.078034 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.078003 5114 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5551d34-16b0-4ed8-97e1-0d49e3d01096-plugins-config-volume\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:19.079800 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.079769 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5551d34-16b0-4ed8-97e1-0d49e3d01096-kube-api-access-ppcvl" pod "b5551d34-16b0-4ed8-97e1-0d49e3d01096" (UID: "b5551d34-16b0-4ed8-97e1-0d49e3d01096"). InnerVolumeSpecName "kube-api-access-ppcvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:52:19.178971 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.178937 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b82e0d2a-6469-4f21-966a-a2048217f811-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b82e0d2a-6469-4f21-966a-a2048217f811-plugins-config-volume\") pod \"b82e0d2a-6469-4f21-966a-a2048217f811\" (UID: \"b82e0d2a-6469-4f21-966a-a2048217f811\") " Nov 02 23:52:19.179180 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.178981 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b82e0d2a-6469-4f21-966a-a2048217f811-kube-api-access-hlrl2\" (UniqueName: \"kubernetes.io/projected/b82e0d2a-6469-4f21-966a-a2048217f811-kube-api-access-hlrl2\") pod \"b82e0d2a-6469-4f21-966a-a2048217f811\" (UID: \"b82e0d2a-6469-4f21-966a-a2048217f811\") " Nov 02 23:52:19.179401 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.179374 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b82e0d2a-6469-4f21-966a-a2048217f811-plugins-config-volume" pod "b82e0d2a-6469-4f21-966a-a2048217f811" (UID: "b82e0d2a-6469-4f21-966a-a2048217f811"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:52:19.179687 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.179671 5114 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b82e0d2a-6469-4f21-966a-a2048217f811-plugins-config-volume\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:19.179740 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.179692 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppcvl\" (UniqueName: \"kubernetes.io/projected/b5551d34-16b0-4ed8-97e1-0d49e3d01096-kube-api-access-ppcvl\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:19.181314 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.181286 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b82e0d2a-6469-4f21-966a-a2048217f811-kube-api-access-hlrl2" pod "b82e0d2a-6469-4f21-966a-a2048217f811" (UID: "b82e0d2a-6469-4f21-966a-a2048217f811"). InnerVolumeSpecName "kube-api-access-hlrl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:52:19.280437 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.280337 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlrl2\" (UniqueName: \"kubernetes.io/projected/b82e0d2a-6469-4f21-966a-a2048217f811-kube-api-access-hlrl2\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:19.722972 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.722938 5114 generic.go:356] "Generic (PLEG): container finished" podID="b5551d34-16b0-4ed8-97e1-0d49e3d01096" containerID="5821b30b00386914ff3b1dc64b9fb4c75b6fca2e7a21b841ea35d4fb3738403f" exitCode=0 Nov 02 23:52:19.723430 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.722985 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg" Nov 02 23:52:19.723430 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.723023 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg" event={"ID":"b5551d34-16b0-4ed8-97e1-0d49e3d01096","Type":"ContainerDied","Data":"5821b30b00386914ff3b1dc64b9fb4c75b6fca2e7a21b841ea35d4fb3738403f"} Nov 02 23:52:19.723430 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.723089 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg" event={"ID":"b5551d34-16b0-4ed8-97e1-0d49e3d01096","Type":"ContainerDied","Data":"10165755abfbe038275e844d85c78abb5fac74b45b4a66f2f1f2892349fc8839"} Nov 02 23:52:19.723430 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.723124 5114 scope.go:122] "RemoveContainer" containerID="5821b30b00386914ff3b1dc64b9fb4c75b6fca2e7a21b841ea35d4fb3738403f" Nov 02 23:52:19.724623 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.724409 5114 generic.go:356] "Generic (PLEG): container finished" podID="b82e0d2a-6469-4f21-966a-a2048217f811" containerID="f4c31ccedbc2ec044cb766737e85e285c5b8c5173036a4512f37f489e6490654" exitCode=0 Nov 02 23:52:19.724623 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.724459 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq" Nov 02 23:52:19.724623 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.724525 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq" event={"ID":"b82e0d2a-6469-4f21-966a-a2048217f811","Type":"ContainerDied","Data":"f4c31ccedbc2ec044cb766737e85e285c5b8c5173036a4512f37f489e6490654"} Nov 02 23:52:19.724623 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.724581 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq" event={"ID":"b82e0d2a-6469-4f21-966a-a2048217f811","Type":"ContainerDied","Data":"387e314403d38449d4358592d25ac29e04fdcd9e807fa805bec4238992f86e91"} Nov 02 23:52:19.735064 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.735044 5114 scope.go:122] "RemoveContainer" containerID="5821b30b00386914ff3b1dc64b9fb4c75b6fca2e7a21b841ea35d4fb3738403f" Nov 02 23:52:19.735445 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:52:19.735424 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5821b30b00386914ff3b1dc64b9fb4c75b6fca2e7a21b841ea35d4fb3738403f\": container with ID starting with 5821b30b00386914ff3b1dc64b9fb4c75b6fca2e7a21b841ea35d4fb3738403f not found: ID does not exist" containerID="5821b30b00386914ff3b1dc64b9fb4c75b6fca2e7a21b841ea35d4fb3738403f" Nov 02 23:52:19.735517 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.735459 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5821b30b00386914ff3b1dc64b9fb4c75b6fca2e7a21b841ea35d4fb3738403f"} err="failed to get container status \"5821b30b00386914ff3b1dc64b9fb4c75b6fca2e7a21b841ea35d4fb3738403f\": rpc error: code = NotFound desc = could not find container \"5821b30b00386914ff3b1dc64b9fb4c75b6fca2e7a21b841ea35d4fb3738403f\": container with ID starting with 5821b30b00386914ff3b1dc64b9fb4c75b6fca2e7a21b841ea35d4fb3738403f not found: ID does not exist" Nov 02 23:52:19.735517 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.735505 5114 scope.go:122] "RemoveContainer" containerID="f4c31ccedbc2ec044cb766737e85e285c5b8c5173036a4512f37f489e6490654" Nov 02 23:52:19.745645 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.745615 5114 scope.go:122] "RemoveContainer" containerID="f4c31ccedbc2ec044cb766737e85e285c5b8c5173036a4512f37f489e6490654" Nov 02 23:52:19.745969 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:52:19.745950 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4c31ccedbc2ec044cb766737e85e285c5b8c5173036a4512f37f489e6490654\": container with ID starting with f4c31ccedbc2ec044cb766737e85e285c5b8c5173036a4512f37f489e6490654 not found: ID does not exist" containerID="f4c31ccedbc2ec044cb766737e85e285c5b8c5173036a4512f37f489e6490654" Nov 02 23:52:19.746042 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.745977 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c31ccedbc2ec044cb766737e85e285c5b8c5173036a4512f37f489e6490654"} err="failed to get container status \"f4c31ccedbc2ec044cb766737e85e285c5b8c5173036a4512f37f489e6490654\": rpc error: code = NotFound desc = could not find container \"f4c31ccedbc2ec044cb766737e85e285c5b8c5173036a4512f37f489e6490654\": container with ID starting with f4c31ccedbc2ec044cb766737e85e285c5b8c5173036a4512f37f489e6490654 not found: ID does not exist" Nov 02 23:52:19.886586 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.886549 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg"] Nov 02 23:52:19.936192 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.936153 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1498df-74bbdd69c9-sqscg"] Nov 02 23:52:19.998321 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:19.998227 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq"] Nov 02 23:52:20.003948 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:20.003915 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1498df-6bf4c54b8c-rn5sq"] Nov 02 23:52:20.794004 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:20.793974 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5551d34-16b0-4ed8-97e1-0d49e3d01096" path="/var/lib/kubelet/pods/b5551d34-16b0-4ed8-97e1-0d49e3d01096/volumes" Nov 02 23:52:20.794455 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:20.794439 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b82e0d2a-6469-4f21-966a-a2048217f811" path="/var/lib/kubelet/pods/b82e0d2a-6469-4f21-966a-a2048217f811/volumes" Nov 02 23:52:37.584509 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:52:37.584445 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:52:47.680397 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:47.680358 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g"] Nov 02 23:52:47.729158 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:47.729125 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f49ab-qnd9g"] Nov 02 23:52:48.793386 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:48.793348 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="708cbf69-febc-4f09-a2d6-518e77977bf7" path="/var/lib/kubelet/pods/708cbf69-febc-4f09-a2d6-518e77977bf7/volumes" Nov 02 23:52:56.318647 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.318603 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9"] Nov 02 23:52:56.319197 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.319086 5114 cpu_manager.go:414] "containerMap: removing container" podUID="b82e0d2a-6469-4f21-966a-a2048217f811" containerName="payload-pre-processing" Nov 02 23:52:56.319197 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.319097 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b82e0d2a-6469-4f21-966a-a2048217f811" containerName="payload-pre-processing" Nov 02 23:52:56.319197 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.319133 5114 cpu_manager.go:414] "containerMap: removing container" podUID="b5551d34-16b0-4ed8-97e1-0d49e3d01096" containerName="payload-processing" Nov 02 23:52:56.319197 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.319138 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5551d34-16b0-4ed8-97e1-0d49e3d01096" containerName="payload-processing" Nov 02 23:52:56.319352 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.319213 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5551d34-16b0-4ed8-97e1-0d49e3d01096" containerName="payload-processing" Nov 02 23:52:56.319352 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.319222 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="b82e0d2a-6469-4f21-966a-a2048217f811" containerName="payload-pre-processing" Nov 02 23:52:56.322440 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.322414 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9" Nov 02 23:52:56.378560 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.378528 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3cd3d8e73f49\"" Nov 02 23:52:56.379037 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.378545 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 23:52:56.409699 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.409665 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a20417-e7a3-412d-ba1b-1a9bd74af91d-plugins-config-volume\") pod \"payload-processing-external-model-67c98f8797-8vbn9\" (UID: \"96a20417-e7a3-412d-ba1b-1a9bd74af91d\") " pod="models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9" Nov 02 23:52:56.409699 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.409701 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96a20417-e7a3-412d-ba1b-1a9bd74af91d-routing-overlay-volume\") pod \"payload-processing-external-model-67c98f8797-8vbn9\" (UID: \"96a20417-e7a3-412d-ba1b-1a9bd74af91d\") " pod="models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9" Nov 02 23:52:56.409946 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.409729 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/96a20417-e7a3-412d-ba1b-1a9bd74af91d-provider-credentials\") pod \"payload-processing-external-model-67c98f8797-8vbn9\" (UID: \"96a20417-e7a3-412d-ba1b-1a9bd74af91d\") " pod="models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9" Nov 02 23:52:56.409946 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.409847 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r86cv\" (UniqueName: \"kubernetes.io/projected/96a20417-e7a3-412d-ba1b-1a9bd74af91d-kube-api-access-r86cv\") pod \"payload-processing-external-model-67c98f8797-8vbn9\" (UID: \"96a20417-e7a3-412d-ba1b-1a9bd74af91d\") " pod="models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9" Nov 02 23:52:56.423714 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.423679 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 23:52:56.423714 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.423679 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 23:52:56.423714 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.423711 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 23:52:56.426087 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.426062 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9"] Nov 02 23:52:56.510692 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.510661 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a20417-e7a3-412d-ba1b-1a9bd74af91d-plugins-config-volume\") pod \"payload-processing-external-model-67c98f8797-8vbn9\" (UID: \"96a20417-e7a3-412d-ba1b-1a9bd74af91d\") " pod="models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9" Nov 02 23:52:56.510692 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.510694 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96a20417-e7a3-412d-ba1b-1a9bd74af91d-routing-overlay-volume\") pod \"payload-processing-external-model-67c98f8797-8vbn9\" (UID: \"96a20417-e7a3-412d-ba1b-1a9bd74af91d\") " pod="models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9" Nov 02 23:52:56.510977 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.510714 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/96a20417-e7a3-412d-ba1b-1a9bd74af91d-provider-credentials\") pod \"payload-processing-external-model-67c98f8797-8vbn9\" (UID: \"96a20417-e7a3-412d-ba1b-1a9bd74af91d\") " pod="models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9" Nov 02 23:52:56.510977 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.510746 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r86cv\" (UniqueName: \"kubernetes.io/projected/96a20417-e7a3-412d-ba1b-1a9bd74af91d-kube-api-access-r86cv\") pod \"payload-processing-external-model-67c98f8797-8vbn9\" (UID: \"96a20417-e7a3-412d-ba1b-1a9bd74af91d\") " pod="models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9" Nov 02 23:52:56.511566 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.511541 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96a20417-e7a3-412d-ba1b-1a9bd74af91d-routing-overlay-volume\") pod \"payload-processing-external-model-67c98f8797-8vbn9\" (UID: \"96a20417-e7a3-412d-ba1b-1a9bd74af91d\") " pod="models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9" Nov 02 23:52:56.511660 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.511609 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a20417-e7a3-412d-ba1b-1a9bd74af91d-plugins-config-volume\") pod \"payload-processing-external-model-67c98f8797-8vbn9\" (UID: \"96a20417-e7a3-412d-ba1b-1a9bd74af91d\") " pod="models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9" Nov 02 23:52:56.513436 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.513414 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/96a20417-e7a3-412d-ba1b-1a9bd74af91d-provider-credentials\") pod \"payload-processing-external-model-67c98f8797-8vbn9\" (UID: \"96a20417-e7a3-412d-ba1b-1a9bd74af91d\") " pod="models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9" Nov 02 23:52:56.571577 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.571474 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r86cv\" (UniqueName: \"kubernetes.io/projected/96a20417-e7a3-412d-ba1b-1a9bd74af91d-kube-api-access-r86cv\") pod \"payload-processing-external-model-67c98f8797-8vbn9\" (UID: \"96a20417-e7a3-412d-ba1b-1a9bd74af91d\") " pod="models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9" Nov 02 23:52:56.636083 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:56.636041 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9" Nov 02 23:52:57.021042 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:57.021009 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9"] Nov 02 23:52:57.023291 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:52:57.023255 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a20417_e7a3_412d_ba1b_1a9bd74af91d.slice/crio-5abc71d1ddb0d7f60f3b3acbce2856565544f5606b11d9a947f31789e517039a WatchSource:0}: Error finding container 5abc71d1ddb0d7f60f3b3acbce2856565544f5606b11d9a947f31789e517039a: Status 404 returned error can't find the container with id 5abc71d1ddb0d7f60f3b3acbce2856565544f5606b11d9a947f31789e517039a Nov 02 23:52:57.964886 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:57.964840 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9" event={"ID":"96a20417-e7a3-412d-ba1b-1a9bd74af91d","Type":"ContainerStarted","Data":"b8e54e0020b0b9c4346ba79786f64c606f2f7b5655675b2414248d15ca5fdbfa"} Nov 02 23:52:57.965291 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:57.965020 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9" Nov 02 23:52:57.965291 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:57.965050 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9" event={"ID":"96a20417-e7a3-412d-ba1b-1a9bd74af91d","Type":"ContainerStarted","Data":"5abc71d1ddb0d7f60f3b3acbce2856565544f5606b11d9a947f31789e517039a"} Nov 02 23:52:58.083561 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:52:58.083501 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9" podStartSLOduration=2.083473684 podStartE2EDuration="2.083473684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:52:58.08098618 +0000 UTC m=+3853.434019556" watchObservedRunningTime="2026-11-02 23:52:58.083473684 +0000 UTC m=+3853.436507059" Nov 02 23:53:07.585028 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:53:07.584984 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:53:08.973960 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:08.973927 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9" Nov 02 23:53:37.584729 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:53:37.584646 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:53:38.163837 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:38.163798 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e130b395-v4lgm"] Nov 02 23:53:38.167786 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:38.167760 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e130b395-v4lgm" Nov 02 23:53:38.219759 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:38.219717 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:53:38.269290 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:38.269258 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:53:38.271553 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:38.271528 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e130b395-v4lgm"] Nov 02 23:53:38.307676 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:38.307621 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c9c6f1-eb52-423a-9284-496cfbbb7cad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e130b395-v4lgm\" (UID: \"d7c9c6f1-eb52-423a-9284-496cfbbb7cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e130b395-v4lgm" Nov 02 23:53:38.307676 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:38.307682 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb7hx\" (UniqueName: \"kubernetes.io/projected/d7c9c6f1-eb52-423a-9284-496cfbbb7cad-kube-api-access-wb7hx\") pod \"maas-api-revoke-sub-e130b395-v4lgm\" (UID: \"d7c9c6f1-eb52-423a-9284-496cfbbb7cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e130b395-v4lgm" Nov 02 23:53:38.307912 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:38.307827 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c9c6f1-eb52-423a-9284-496cfbbb7cad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e130b395-v4lgm\" (UID: \"d7c9c6f1-eb52-423a-9284-496cfbbb7cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e130b395-v4lgm" Nov 02 23:53:38.408878 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:38.408835 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c9c6f1-eb52-423a-9284-496cfbbb7cad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e130b395-v4lgm\" (UID: \"d7c9c6f1-eb52-423a-9284-496cfbbb7cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e130b395-v4lgm" Nov 02 23:53:38.408878 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:38.408884 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb7hx\" (UniqueName: \"kubernetes.io/projected/d7c9c6f1-eb52-423a-9284-496cfbbb7cad-kube-api-access-wb7hx\") pod \"maas-api-revoke-sub-e130b395-v4lgm\" (UID: \"d7c9c6f1-eb52-423a-9284-496cfbbb7cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e130b395-v4lgm" Nov 02 23:53:38.409119 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:38.408939 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c9c6f1-eb52-423a-9284-496cfbbb7cad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e130b395-v4lgm\" (UID: \"d7c9c6f1-eb52-423a-9284-496cfbbb7cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e130b395-v4lgm" Nov 02 23:53:38.409678 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:38.409650 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c9c6f1-eb52-423a-9284-496cfbbb7cad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e130b395-v4lgm\" (UID: \"d7c9c6f1-eb52-423a-9284-496cfbbb7cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e130b395-v4lgm" Nov 02 23:53:38.411492 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:38.411461 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c9c6f1-eb52-423a-9284-496cfbbb7cad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e130b395-v4lgm\" (UID: \"d7c9c6f1-eb52-423a-9284-496cfbbb7cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e130b395-v4lgm" Nov 02 23:53:38.474626 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:38.474544 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb7hx\" (UniqueName: \"kubernetes.io/projected/d7c9c6f1-eb52-423a-9284-496cfbbb7cad-kube-api-access-wb7hx\") pod \"maas-api-revoke-sub-e130b395-v4lgm\" (UID: \"d7c9c6f1-eb52-423a-9284-496cfbbb7cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e130b395-v4lgm" Nov 02 23:53:38.482373 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:38.482349 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e130b395-v4lgm" Nov 02 23:53:38.859388 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:38.859361 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e130b395-v4lgm"] Nov 02 23:53:38.861660 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:53:38.861625 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c9c6f1_eb52_423a_9284_496cfbbb7cad.slice/crio-293f55b2c7a0dce5ee2b2bcadb781e7948cf5f8b6459cb9c5ace4c5f75a65156 WatchSource:0}: Error finding container 293f55b2c7a0dce5ee2b2bcadb781e7948cf5f8b6459cb9c5ace4c5f75a65156: Status 404 returned error can't find the container with id 293f55b2c7a0dce5ee2b2bcadb781e7948cf5f8b6459cb9c5ace4c5f75a65156 Nov 02 23:53:39.195720 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:39.195680 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e130b395-v4lgm" event={"ID":"d7c9c6f1-eb52-423a-9284-496cfbbb7cad","Type":"ContainerStarted","Data":"293f55b2c7a0dce5ee2b2bcadb781e7948cf5f8b6459cb9c5ace4c5f75a65156"} Nov 02 23:53:40.126163 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:40.126064 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j"] Nov 02 23:53:40.126712 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:40.126300 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" podUID="d0400368-a485-486f-b433-3c456aaddff9" containerName="istio-proxy" containerID="cri-o://6f8ab0398ebb42b754c3710d7442a4686550e44a5e2a443cf23ff114c5108ae9" gracePeriod=30 Nov 02 23:53:40.206706 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:40.206664 5114 generic.go:356] "Generic (PLEG): container finished" podID="d7c9c6f1-eb52-423a-9284-496cfbbb7cad" containerID="ebbab4d6411fb6f63e6f73121899979b93cfcb4a74df2384eb7670bf528a4975" exitCode=0 Nov 02 23:53:40.206918 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:40.206753 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e130b395-v4lgm" event={"ID":"d7c9c6f1-eb52-423a-9284-496cfbbb7cad","Type":"ContainerDied","Data":"ebbab4d6411fb6f63e6f73121899979b93cfcb4a74df2384eb7670bf528a4975"} Nov 02 23:53:41.375704 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:41.375670 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e130b395-v4lgm" Nov 02 23:53:41.545344 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:41.545297 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7c9c6f1-eb52-423a-9284-496cfbbb7cad-kube-api-access-wb7hx\" (UniqueName: \"kubernetes.io/projected/d7c9c6f1-eb52-423a-9284-496cfbbb7cad-kube-api-access-wb7hx\") pod \"d7c9c6f1-eb52-423a-9284-496cfbbb7cad\" (UID: \"d7c9c6f1-eb52-423a-9284-496cfbbb7cad\") " Nov 02 23:53:41.545554 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:41.545376 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7c9c6f1-eb52-423a-9284-496cfbbb7cad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c9c6f1-eb52-423a-9284-496cfbbb7cad-maas-api-xks-ca\") pod \"d7c9c6f1-eb52-423a-9284-496cfbbb7cad\" (UID: \"d7c9c6f1-eb52-423a-9284-496cfbbb7cad\") " Nov 02 23:53:41.545622 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:41.545548 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7c9c6f1-eb52-423a-9284-496cfbbb7cad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c9c6f1-eb52-423a-9284-496cfbbb7cad-maas-api-openshift-ca\") pod \"d7c9c6f1-eb52-423a-9284-496cfbbb7cad\" (UID: \"d7c9c6f1-eb52-423a-9284-496cfbbb7cad\") " Nov 02 23:53:41.545895 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:41.545871 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7c9c6f1-eb52-423a-9284-496cfbbb7cad-maas-api-openshift-ca" pod "d7c9c6f1-eb52-423a-9284-496cfbbb7cad" (UID: "d7c9c6f1-eb52-423a-9284-496cfbbb7cad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:53:41.548511 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:41.547807 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c9c6f1-eb52-423a-9284-496cfbbb7cad-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:41.557338 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:41.557294 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c9c6f1-eb52-423a-9284-496cfbbb7cad-kube-api-access-wb7hx" pod "d7c9c6f1-eb52-423a-9284-496cfbbb7cad" (UID: "d7c9c6f1-eb52-423a-9284-496cfbbb7cad"). InnerVolumeSpecName "kube-api-access-wb7hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:53:41.559303 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:41.559268 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7c9c6f1-eb52-423a-9284-496cfbbb7cad-maas-api-xks-ca" pod "d7c9c6f1-eb52-423a-9284-496cfbbb7cad" (UID: "d7c9c6f1-eb52-423a-9284-496cfbbb7cad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:53:41.648856 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:41.648806 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb7hx\" (UniqueName: \"kubernetes.io/projected/d7c9c6f1-eb52-423a-9284-496cfbbb7cad-kube-api-access-wb7hx\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:41.648856 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:41.648846 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c9c6f1-eb52-423a-9284-496cfbbb7cad-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:42.220592 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:42.220560 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e130b395-v4lgm" Nov 02 23:53:42.220765 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:42.220602 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e130b395-v4lgm" event={"ID":"d7c9c6f1-eb52-423a-9284-496cfbbb7cad","Type":"ContainerDied","Data":"293f55b2c7a0dce5ee2b2bcadb781e7948cf5f8b6459cb9c5ace4c5f75a65156"} Nov 02 23:53:42.220765 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:42.220655 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="293f55b2c7a0dce5ee2b2bcadb781e7948cf5f8b6459cb9c5ace4c5f75a65156" Nov 02 23:53:45.405635 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.405560 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:53:45.492935 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.492900 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-workload-certs\") pod \"d0400368-a485-486f-b433-3c456aaddff9\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " Nov 02 23:53:45.493148 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.492967 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0400368-a485-486f-b433-3c456aaddff9-kube-api-access-gfb54\" (UniqueName: \"kubernetes.io/projected/d0400368-a485-486f-b433-3c456aaddff9-kube-api-access-gfb54\") pod \"d0400368-a485-486f-b433-3c456aaddff9\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " Nov 02 23:53:45.493148 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.493021 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0400368-a485-486f-b433-3c456aaddff9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0400368-a485-486f-b433-3c456aaddff9-istio-podinfo\") pod \"d0400368-a485-486f-b433-3c456aaddff9\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " Nov 02 23:53:45.493148 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.493055 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-workload-socket\") pod \"d0400368-a485-486f-b433-3c456aaddff9\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " Nov 02 23:53:45.493320 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.493192 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0400368-a485-486f-b433-3c456aaddff9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0400368-a485-486f-b433-3c456aaddff9-istiod-ca-cert\") pod \"d0400368-a485-486f-b433-3c456aaddff9\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " Nov 02 23:53:45.493320 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.493240 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-credential-socket\") pod \"d0400368-a485-486f-b433-3c456aaddff9\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " Nov 02 23:53:45.493320 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.493285 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-istio-envoy\") pod \"d0400368-a485-486f-b433-3c456aaddff9\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " Nov 02 23:53:45.493320 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.493284 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-workload-certs" pod "d0400368-a485-486f-b433-3c456aaddff9" (UID: "d0400368-a485-486f-b433-3c456aaddff9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:53:45.493582 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.493333 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-istio-data\") pod \"d0400368-a485-486f-b433-3c456aaddff9\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " Nov 02 23:53:45.493582 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.493389 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0400368-a485-486f-b433-3c456aaddff9-istio-token\" (UniqueName: \"kubernetes.io/projected/d0400368-a485-486f-b433-3c456aaddff9-istio-token\") pod \"d0400368-a485-486f-b433-3c456aaddff9\" (UID: \"d0400368-a485-486f-b433-3c456aaddff9\") " Nov 02 23:53:45.493582 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.493421 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-workload-socket" pod "d0400368-a485-486f-b433-3c456aaddff9" (UID: "d0400368-a485-486f-b433-3c456aaddff9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:53:45.493757 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.493614 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-credential-socket" pod "d0400368-a485-486f-b433-3c456aaddff9" (UID: "d0400368-a485-486f-b433-3c456aaddff9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:53:45.493757 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.493641 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0400368-a485-486f-b433-3c456aaddff9-istiod-ca-cert" pod "d0400368-a485-486f-b433-3c456aaddff9" (UID: "d0400368-a485-486f-b433-3c456aaddff9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:53:45.493999 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.493914 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:45.493999 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.493944 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:45.493999 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.493962 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0400368-a485-486f-b433-3c456aaddff9-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:45.493999 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.493977 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:45.494398 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.494246 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-istio-data" pod "d0400368-a485-486f-b433-3c456aaddff9" (UID: "d0400368-a485-486f-b433-3c456aaddff9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:53:45.495672 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.495640 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-istio-envoy" pod "d0400368-a485-486f-b433-3c456aaddff9" (UID: "d0400368-a485-486f-b433-3c456aaddff9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:53:45.495762 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.495678 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0400368-a485-486f-b433-3c456aaddff9-istio-podinfo" pod "d0400368-a485-486f-b433-3c456aaddff9" (UID: "d0400368-a485-486f-b433-3c456aaddff9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:53:45.495762 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.495743 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0400368-a485-486f-b433-3c456aaddff9-kube-api-access-gfb54" pod "d0400368-a485-486f-b433-3c456aaddff9" (UID: "d0400368-a485-486f-b433-3c456aaddff9"). InnerVolumeSpecName "kube-api-access-gfb54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:53:45.495846 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.495767 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0400368-a485-486f-b433-3c456aaddff9-istio-token" pod "d0400368-a485-486f-b433-3c456aaddff9" (UID: "d0400368-a485-486f-b433-3c456aaddff9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:53:45.595096 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.595055 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:45.595096 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.595090 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0400368-a485-486f-b433-3c456aaddff9-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:45.595096 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.595099 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0400368-a485-486f-b433-3c456aaddff9-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:45.595364 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.595108 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfb54\" (UniqueName: \"kubernetes.io/projected/d0400368-a485-486f-b433-3c456aaddff9-kube-api-access-gfb54\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:45.595364 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.595119 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0400368-a485-486f-b433-3c456aaddff9-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:45.961112 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:45.961073 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69"] Nov 02 23:53:46.035234 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:46.035202 5114 scope.go:122] "RemoveContainer" containerID="fc245d2caaf0f037230538c9d8d2279e4659dfaa2717bde9a684b8efeb6526ec" Nov 02 23:53:46.045807 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:46.045786 5114 scope.go:122] "RemoveContainer" containerID="7bb850e4a88cf684fae83c5159cbecd1e9b3c0c3a95d5407a969ea0156cc5623" Nov 02 23:53:46.074078 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:46.074043 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v"] Nov 02 23:53:46.248914 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:46.248817 5114 generic.go:356] "Generic (PLEG): container finished" podID="d0400368-a485-486f-b433-3c456aaddff9" containerID="6f8ab0398ebb42b754c3710d7442a4686550e44a5e2a443cf23ff114c5108ae9" exitCode=0 Nov 02 23:53:46.248914 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:46.248880 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" Nov 02 23:53:46.249171 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:46.248903 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" event={"ID":"d0400368-a485-486f-b433-3c456aaddff9","Type":"ContainerDied","Data":"6f8ab0398ebb42b754c3710d7442a4686550e44a5e2a443cf23ff114c5108ae9"} Nov 02 23:53:46.249171 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:46.248975 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j" event={"ID":"d0400368-a485-486f-b433-3c456aaddff9","Type":"ContainerDied","Data":"56a0379782b03853f321175f7bbd43daca543efe40c72c5cec68690223713878"} Nov 02 23:53:46.249171 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:46.249003 5114 scope.go:122] "RemoveContainer" containerID="6f8ab0398ebb42b754c3710d7442a4686550e44a5e2a443cf23ff114c5108ae9" Nov 02 23:53:46.262003 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:46.261980 5114 scope.go:122] "RemoveContainer" containerID="6f8ab0398ebb42b754c3710d7442a4686550e44a5e2a443cf23ff114c5108ae9" Nov 02 23:53:46.262373 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:53:46.262350 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f8ab0398ebb42b754c3710d7442a4686550e44a5e2a443cf23ff114c5108ae9\": container with ID starting with 6f8ab0398ebb42b754c3710d7442a4686550e44a5e2a443cf23ff114c5108ae9 not found: ID does not exist" containerID="6f8ab0398ebb42b754c3710d7442a4686550e44a5e2a443cf23ff114c5108ae9" Nov 02 23:53:46.262472 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:46.262384 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f8ab0398ebb42b754c3710d7442a4686550e44a5e2a443cf23ff114c5108ae9"} err="failed to get container status \"6f8ab0398ebb42b754c3710d7442a4686550e44a5e2a443cf23ff114c5108ae9\": rpc error: code = NotFound desc = could not find container \"6f8ab0398ebb42b754c3710d7442a4686550e44a5e2a443cf23ff114c5108ae9\": container with ID starting with 6f8ab0398ebb42b754c3710d7442a4686550e44a5e2a443cf23ff114c5108ae9 not found: ID does not exist" Nov 02 23:53:46.482515 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:46.482463 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j"] Nov 02 23:53:46.488903 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:46.488873 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1498df-openshift-default-5848944d9c-djq5j"] Nov 02 23:53:46.793575 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:46.793541 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0400368-a485-486f-b433-3c456aaddff9" path="/var/lib/kubelet/pods/d0400368-a485-486f-b433-3c456aaddff9/volumes" Nov 02 23:53:50.380584 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:50.380548 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9"] Nov 02 23:53:50.961475 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:50.961397 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69" podUID="69bf017e-78f8-46c9-95eb-08647f5d1788" containerName="payload-processing" containerID="cri-o://93ced4f4c566b8fe24e1aa9944c300f2e6306629a8df91d1a49b22b7a8301653" gracePeriod=25 Nov 02 23:53:51.074609 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.074565 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v" podUID="2775490f-e402-4fa8-af5a-df684e8a0198" containerName="payload-pre-processing" containerID="cri-o://9fd3f2ff0dc721a9c01bbe5c01e6d798eae9e3b6b169064c5a909ddcc646fa77" gracePeriod=25 Nov 02 23:53:51.251193 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.251163 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69" Nov 02 23:53:51.283245 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.283209 5114 generic.go:356] "Generic (PLEG): container finished" podID="2775490f-e402-4fa8-af5a-df684e8a0198" containerID="9fd3f2ff0dc721a9c01bbe5c01e6d798eae9e3b6b169064c5a909ddcc646fa77" exitCode=0 Nov 02 23:53:51.283444 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.283281 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v" event={"ID":"2775490f-e402-4fa8-af5a-df684e8a0198","Type":"ContainerDied","Data":"9fd3f2ff0dc721a9c01bbe5c01e6d798eae9e3b6b169064c5a909ddcc646fa77"} Nov 02 23:53:51.285065 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.285037 5114 generic.go:356] "Generic (PLEG): container finished" podID="69bf017e-78f8-46c9-95eb-08647f5d1788" containerID="93ced4f4c566b8fe24e1aa9944c300f2e6306629a8df91d1a49b22b7a8301653" exitCode=0 Nov 02 23:53:51.285215 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.285097 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69" Nov 02 23:53:51.285215 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.285111 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69" event={"ID":"69bf017e-78f8-46c9-95eb-08647f5d1788","Type":"ContainerDied","Data":"93ced4f4c566b8fe24e1aa9944c300f2e6306629a8df91d1a49b22b7a8301653"} Nov 02 23:53:51.285215 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.285160 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69" event={"ID":"69bf017e-78f8-46c9-95eb-08647f5d1788","Type":"ContainerDied","Data":"aab6e4f08119b318a47214f3750166f91210dc5ef59b7d121f259653b13b337a"} Nov 02 23:53:51.285215 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.285191 5114 scope.go:122] "RemoveContainer" containerID="93ced4f4c566b8fe24e1aa9944c300f2e6306629a8df91d1a49b22b7a8301653" Nov 02 23:53:51.319781 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.319724 5114 scope.go:122] "RemoveContainer" containerID="93ced4f4c566b8fe24e1aa9944c300f2e6306629a8df91d1a49b22b7a8301653" Nov 02 23:53:51.320563 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:53:51.320454 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93ced4f4c566b8fe24e1aa9944c300f2e6306629a8df91d1a49b22b7a8301653\": container with ID starting with 93ced4f4c566b8fe24e1aa9944c300f2e6306629a8df91d1a49b22b7a8301653 not found: ID does not exist" containerID="93ced4f4c566b8fe24e1aa9944c300f2e6306629a8df91d1a49b22b7a8301653" Nov 02 23:53:51.320563 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.320507 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93ced4f4c566b8fe24e1aa9944c300f2e6306629a8df91d1a49b22b7a8301653"} err="failed to get container status \"93ced4f4c566b8fe24e1aa9944c300f2e6306629a8df91d1a49b22b7a8301653\": rpc error: code = NotFound desc = could not find container \"93ced4f4c566b8fe24e1aa9944c300f2e6306629a8df91d1a49b22b7a8301653\": container with ID starting with 93ced4f4c566b8fe24e1aa9944c300f2e6306629a8df91d1a49b22b7a8301653 not found: ID does not exist" Nov 02 23:53:51.367427 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.367401 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69bf017e-78f8-46c9-95eb-08647f5d1788-kube-api-access-sb7px\" (UniqueName: \"kubernetes.io/projected/69bf017e-78f8-46c9-95eb-08647f5d1788-kube-api-access-sb7px\") pod \"69bf017e-78f8-46c9-95eb-08647f5d1788\" (UID: \"69bf017e-78f8-46c9-95eb-08647f5d1788\") " Nov 02 23:53:51.367524 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.367455 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69bf017e-78f8-46c9-95eb-08647f5d1788-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69bf017e-78f8-46c9-95eb-08647f5d1788-plugins-config-volume\") pod \"69bf017e-78f8-46c9-95eb-08647f5d1788\" (UID: \"69bf017e-78f8-46c9-95eb-08647f5d1788\") " Nov 02 23:53:51.367933 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.367878 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69bf017e-78f8-46c9-95eb-08647f5d1788-plugins-config-volume" pod "69bf017e-78f8-46c9-95eb-08647f5d1788" (UID: "69bf017e-78f8-46c9-95eb-08647f5d1788"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:53:51.371321 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.371289 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69bf017e-78f8-46c9-95eb-08647f5d1788-kube-api-access-sb7px" pod "69bf017e-78f8-46c9-95eb-08647f5d1788" (UID: "69bf017e-78f8-46c9-95eb-08647f5d1788"). InnerVolumeSpecName "kube-api-access-sb7px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:53:51.383051 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.383022 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v" Nov 02 23:53:51.469043 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.468953 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2775490f-e402-4fa8-af5a-df684e8a0198-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2775490f-e402-4fa8-af5a-df684e8a0198-plugins-config-volume\") pod \"2775490f-e402-4fa8-af5a-df684e8a0198\" (UID: \"2775490f-e402-4fa8-af5a-df684e8a0198\") " Nov 02 23:53:51.469043 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.469020 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2775490f-e402-4fa8-af5a-df684e8a0198-kube-api-access-w42t9\" (UniqueName: \"kubernetes.io/projected/2775490f-e402-4fa8-af5a-df684e8a0198-kube-api-access-w42t9\") pod \"2775490f-e402-4fa8-af5a-df684e8a0198\" (UID: \"2775490f-e402-4fa8-af5a-df684e8a0198\") " Nov 02 23:53:51.469269 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.469245 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2775490f-e402-4fa8-af5a-df684e8a0198-plugins-config-volume" pod "2775490f-e402-4fa8-af5a-df684e8a0198" (UID: "2775490f-e402-4fa8-af5a-df684e8a0198"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:53:51.469521 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.469473 5114 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2775490f-e402-4fa8-af5a-df684e8a0198-plugins-config-volume\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:51.469580 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.469525 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb7px\" (UniqueName: \"kubernetes.io/projected/69bf017e-78f8-46c9-95eb-08647f5d1788-kube-api-access-sb7px\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:51.469580 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.469535 5114 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69bf017e-78f8-46c9-95eb-08647f5d1788-plugins-config-volume\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:51.471454 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.471430 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2775490f-e402-4fa8-af5a-df684e8a0198-kube-api-access-w42t9" pod "2775490f-e402-4fa8-af5a-df684e8a0198" (UID: "2775490f-e402-4fa8-af5a-df684e8a0198"). InnerVolumeSpecName "kube-api-access-w42t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:53:51.570187 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.570145 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w42t9\" (UniqueName: \"kubernetes.io/projected/2775490f-e402-4fa8-af5a-df684e8a0198-kube-api-access-w42t9\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:51.704824 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.704788 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69"] Nov 02 23:53:51.708312 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:51.708281 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-056584-5f944c648f-t5s69"] Nov 02 23:53:52.294148 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:52.294110 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v" event={"ID":"2775490f-e402-4fa8-af5a-df684e8a0198","Type":"ContainerDied","Data":"4353549857b58872aec49287b81bff2c32b7b8d1f7f7974bf3d7a8e76aea1243"} Nov 02 23:53:52.294148 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:52.294146 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v" Nov 02 23:53:52.294422 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:52.294184 5114 scope.go:122] "RemoveContainer" containerID="9fd3f2ff0dc721a9c01bbe5c01e6d798eae9e3b6b169064c5a909ddcc646fa77" Nov 02 23:53:52.456919 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:52.456884 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v"] Nov 02 23:53:52.506581 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:52.506538 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-056584-8f88f7755-42h9v"] Nov 02 23:53:52.793823 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:52.793787 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2775490f-e402-4fa8-af5a-df684e8a0198" path="/var/lib/kubelet/pods/2775490f-e402-4fa8-af5a-df684e8a0198/volumes" Nov 02 23:53:52.794386 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:52.794370 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69bf017e-78f8-46c9-95eb-08647f5d1788" path="/var/lib/kubelet/pods/69bf017e-78f8-46c9-95eb-08647f5d1788/volumes" Nov 02 23:53:55.381937 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:55.381867 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9" podUID="96a20417-e7a3-412d-ba1b-1a9bd74af91d" containerName="payload-processing" containerID="cri-o://b8e54e0020b0b9c4346ba79786f64c606f2f7b5655675b2414248d15ca5fdbfa" gracePeriod=25 Nov 02 23:53:55.694511 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:55.694457 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9" Nov 02 23:53:55.835666 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:55.833990 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96a20417-e7a3-412d-ba1b-1a9bd74af91d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96a20417-e7a3-412d-ba1b-1a9bd74af91d-routing-overlay-volume\") pod \"96a20417-e7a3-412d-ba1b-1a9bd74af91d\" (UID: \"96a20417-e7a3-412d-ba1b-1a9bd74af91d\") " Nov 02 23:53:55.835666 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:55.834070 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96a20417-e7a3-412d-ba1b-1a9bd74af91d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a20417-e7a3-412d-ba1b-1a9bd74af91d-plugins-config-volume\") pod \"96a20417-e7a3-412d-ba1b-1a9bd74af91d\" (UID: \"96a20417-e7a3-412d-ba1b-1a9bd74af91d\") " Nov 02 23:53:55.835666 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:55.834144 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a20417-e7a3-412d-ba1b-1a9bd74af91d-kube-api-access-r86cv\" (UniqueName: \"kubernetes.io/projected/96a20417-e7a3-412d-ba1b-1a9bd74af91d-kube-api-access-r86cv\") pod \"96a20417-e7a3-412d-ba1b-1a9bd74af91d\" (UID: \"96a20417-e7a3-412d-ba1b-1a9bd74af91d\") " Nov 02 23:53:55.835666 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:55.834271 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a20417-e7a3-412d-ba1b-1a9bd74af91d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/96a20417-e7a3-412d-ba1b-1a9bd74af91d-provider-credentials\") pod \"96a20417-e7a3-412d-ba1b-1a9bd74af91d\" (UID: \"96a20417-e7a3-412d-ba1b-1a9bd74af91d\") " Nov 02 23:53:55.835666 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:55.834585 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a20417-e7a3-412d-ba1b-1a9bd74af91d-plugins-config-volume" pod "96a20417-e7a3-412d-ba1b-1a9bd74af91d" (UID: "96a20417-e7a3-412d-ba1b-1a9bd74af91d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:53:55.835666 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:55.834762 5114 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a20417-e7a3-412d-ba1b-1a9bd74af91d-plugins-config-volume\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:55.835666 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:55.835531 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a20417-e7a3-412d-ba1b-1a9bd74af91d-routing-overlay-volume" pod "96a20417-e7a3-412d-ba1b-1a9bd74af91d" (UID: "96a20417-e7a3-412d-ba1b-1a9bd74af91d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:53:55.838260 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:55.838227 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a20417-e7a3-412d-ba1b-1a9bd74af91d-kube-api-access-r86cv" pod "96a20417-e7a3-412d-ba1b-1a9bd74af91d" (UID: "96a20417-e7a3-412d-ba1b-1a9bd74af91d"). InnerVolumeSpecName "kube-api-access-r86cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:53:55.838415 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:55.838270 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a20417-e7a3-412d-ba1b-1a9bd74af91d-provider-credentials" pod "96a20417-e7a3-412d-ba1b-1a9bd74af91d" (UID: "96a20417-e7a3-412d-ba1b-1a9bd74af91d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:53:55.936396 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:55.936307 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r86cv\" (UniqueName: \"kubernetes.io/projected/96a20417-e7a3-412d-ba1b-1a9bd74af91d-kube-api-access-r86cv\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:55.936396 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:55.936341 5114 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/96a20417-e7a3-412d-ba1b-1a9bd74af91d-provider-credentials\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:55.936396 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:55.936350 5114 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96a20417-e7a3-412d-ba1b-1a9bd74af91d-routing-overlay-volume\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:56.334281 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:56.334244 5114 generic.go:356] "Generic (PLEG): container finished" podID="96a20417-e7a3-412d-ba1b-1a9bd74af91d" containerID="b8e54e0020b0b9c4346ba79786f64c606f2f7b5655675b2414248d15ca5fdbfa" exitCode=0 Nov 02 23:53:56.334547 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:56.334300 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9" Nov 02 23:53:56.334547 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:56.334334 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9" event={"ID":"96a20417-e7a3-412d-ba1b-1a9bd74af91d","Type":"ContainerDied","Data":"b8e54e0020b0b9c4346ba79786f64c606f2f7b5655675b2414248d15ca5fdbfa"} Nov 02 23:53:56.334547 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:56.334402 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9" event={"ID":"96a20417-e7a3-412d-ba1b-1a9bd74af91d","Type":"ContainerDied","Data":"5abc71d1ddb0d7f60f3b3acbce2856565544f5606b11d9a947f31789e517039a"} Nov 02 23:53:56.334547 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:56.334438 5114 scope.go:122] "RemoveContainer" containerID="b8e54e0020b0b9c4346ba79786f64c606f2f7b5655675b2414248d15ca5fdbfa" Nov 02 23:53:56.348024 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:56.348001 5114 scope.go:122] "RemoveContainer" containerID="b8e54e0020b0b9c4346ba79786f64c606f2f7b5655675b2414248d15ca5fdbfa" Nov 02 23:53:56.348362 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:53:56.348340 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8e54e0020b0b9c4346ba79786f64c606f2f7b5655675b2414248d15ca5fdbfa\": container with ID starting with b8e54e0020b0b9c4346ba79786f64c606f2f7b5655675b2414248d15ca5fdbfa not found: ID does not exist" containerID="b8e54e0020b0b9c4346ba79786f64c606f2f7b5655675b2414248d15ca5fdbfa" Nov 02 23:53:56.348447 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:56.348373 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8e54e0020b0b9c4346ba79786f64c606f2f7b5655675b2414248d15ca5fdbfa"} err="failed to get container status \"b8e54e0020b0b9c4346ba79786f64c606f2f7b5655675b2414248d15ca5fdbfa\": rpc error: code = NotFound desc = could not find container \"b8e54e0020b0b9c4346ba79786f64c606f2f7b5655675b2414248d15ca5fdbfa\": container with ID starting with b8e54e0020b0b9c4346ba79786f64c606f2f7b5655675b2414248d15ca5fdbfa not found: ID does not exist" Nov 02 23:53:56.501105 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:56.501068 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9"] Nov 02 23:53:56.504458 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:56.504426 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-67c98f8797-8vbn9"] Nov 02 23:53:56.793762 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:56.793729 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96a20417-e7a3-412d-ba1b-1a9bd74af91d" path="/var/lib/kubelet/pods/96a20417-e7a3-412d-ba1b-1a9bd74af91d/volumes" Nov 02 23:53:57.727508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.726266 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-68467d9546-nshjz"] Nov 02 23:53:57.727508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.726911 5114 cpu_manager.go:414] "containerMap: removing container" podUID="96a20417-e7a3-412d-ba1b-1a9bd74af91d" containerName="payload-processing" Nov 02 23:53:57.727508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.726930 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a20417-e7a3-412d-ba1b-1a9bd74af91d" containerName="payload-processing" Nov 02 23:53:57.727508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.726963 5114 cpu_manager.go:414] "containerMap: removing container" podUID="69bf017e-78f8-46c9-95eb-08647f5d1788" containerName="payload-processing" Nov 02 23:53:57.727508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.726972 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69bf017e-78f8-46c9-95eb-08647f5d1788" containerName="payload-processing" Nov 02 23:53:57.727508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.726986 5114 cpu_manager.go:414] "containerMap: removing container" podUID="d0400368-a485-486f-b433-3c456aaddff9" containerName="istio-proxy" Nov 02 23:53:57.727508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.726995 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0400368-a485-486f-b433-3c456aaddff9" containerName="istio-proxy" Nov 02 23:53:57.727508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.727019 5114 cpu_manager.go:414] "containerMap: removing container" podUID="2775490f-e402-4fa8-af5a-df684e8a0198" containerName="payload-pre-processing" Nov 02 23:53:57.727508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.727027 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2775490f-e402-4fa8-af5a-df684e8a0198" containerName="payload-pre-processing" Nov 02 23:53:57.727508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.727051 5114 cpu_manager.go:414] "containerMap: removing container" podUID="d7c9c6f1-eb52-423a-9284-496cfbbb7cad" containerName="revoke-keys" Nov 02 23:53:57.727508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.727062 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7c9c6f1-eb52-423a-9284-496cfbbb7cad" containerName="revoke-keys" Nov 02 23:53:57.727508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.727159 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a20417-e7a3-412d-ba1b-1a9bd74af91d" containerName="payload-processing" Nov 02 23:53:57.727508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.727176 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="69bf017e-78f8-46c9-95eb-08647f5d1788" containerName="payload-processing" Nov 02 23:53:57.727508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.727186 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0400368-a485-486f-b433-3c456aaddff9" containerName="istio-proxy" Nov 02 23:53:57.727508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.727198 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="2775490f-e402-4fa8-af5a-df684e8a0198" containerName="payload-pre-processing" Nov 02 23:53:57.727508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.727206 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0400368-a485-486f-b433-3c456aaddff9" containerName="istio-proxy" Nov 02 23:53:57.727508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.727220 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7c9c6f1-eb52-423a-9284-496cfbbb7cad" containerName="revoke-keys" Nov 02 23:53:57.734595 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.734561 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68467d9546-nshjz" Nov 02 23:53:57.787376 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.787330 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 23:53:57.838830 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.838770 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-58c6f06e6850\"" Nov 02 23:53:57.839054 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.838863 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 23:53:57.839054 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.838977 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 23:53:57.839196 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.839067 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-vfmqn\"" Nov 02 23:53:57.839196 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.839126 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 23:53:57.842378 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.842347 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-68467d9546-nshjz"] Nov 02 23:53:57.858385 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.858332 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab8d9910-c9f8-4b8b-a434-c2355fe39209-provider-credentials\") pod \"payload-processing-external-model-68467d9546-nshjz\" (UID: \"ab8d9910-c9f8-4b8b-a434-c2355fe39209\") " pod="models-as-a-service/payload-processing-external-model-68467d9546-nshjz" Nov 02 23:53:57.858625 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.858395 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8d9910-c9f8-4b8b-a434-c2355fe39209-plugins-config-volume\") pod \"payload-processing-external-model-68467d9546-nshjz\" (UID: \"ab8d9910-c9f8-4b8b-a434-c2355fe39209\") " pod="models-as-a-service/payload-processing-external-model-68467d9546-nshjz" Nov 02 23:53:57.858625 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.858582 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab8d9910-c9f8-4b8b-a434-c2355fe39209-routing-overlay-volume\") pod \"payload-processing-external-model-68467d9546-nshjz\" (UID: \"ab8d9910-c9f8-4b8b-a434-c2355fe39209\") " pod="models-as-a-service/payload-processing-external-model-68467d9546-nshjz" Nov 02 23:53:57.858757 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.858733 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td9fs\" (UniqueName: \"kubernetes.io/projected/ab8d9910-c9f8-4b8b-a434-c2355fe39209-kube-api-access-td9fs\") pod \"payload-processing-external-model-68467d9546-nshjz\" (UID: \"ab8d9910-c9f8-4b8b-a434-c2355fe39209\") " pod="models-as-a-service/payload-processing-external-model-68467d9546-nshjz" Nov 02 23:53:57.960432 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.960395 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab8d9910-c9f8-4b8b-a434-c2355fe39209-provider-credentials\") pod \"payload-processing-external-model-68467d9546-nshjz\" (UID: \"ab8d9910-c9f8-4b8b-a434-c2355fe39209\") " pod="models-as-a-service/payload-processing-external-model-68467d9546-nshjz" Nov 02 23:53:57.960432 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.960439 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8d9910-c9f8-4b8b-a434-c2355fe39209-plugins-config-volume\") pod \"payload-processing-external-model-68467d9546-nshjz\" (UID: \"ab8d9910-c9f8-4b8b-a434-c2355fe39209\") " pod="models-as-a-service/payload-processing-external-model-68467d9546-nshjz" Nov 02 23:53:57.960691 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.960469 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab8d9910-c9f8-4b8b-a434-c2355fe39209-routing-overlay-volume\") pod \"payload-processing-external-model-68467d9546-nshjz\" (UID: \"ab8d9910-c9f8-4b8b-a434-c2355fe39209\") " pod="models-as-a-service/payload-processing-external-model-68467d9546-nshjz" Nov 02 23:53:57.960691 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.960514 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td9fs\" (UniqueName: \"kubernetes.io/projected/ab8d9910-c9f8-4b8b-a434-c2355fe39209-kube-api-access-td9fs\") pod \"payload-processing-external-model-68467d9546-nshjz\" (UID: \"ab8d9910-c9f8-4b8b-a434-c2355fe39209\") " pod="models-as-a-service/payload-processing-external-model-68467d9546-nshjz" Nov 02 23:53:57.961249 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.961209 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab8d9910-c9f8-4b8b-a434-c2355fe39209-routing-overlay-volume\") pod \"payload-processing-external-model-68467d9546-nshjz\" (UID: \"ab8d9910-c9f8-4b8b-a434-c2355fe39209\") " pod="models-as-a-service/payload-processing-external-model-68467d9546-nshjz" Nov 02 23:53:57.961405 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.961264 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8d9910-c9f8-4b8b-a434-c2355fe39209-plugins-config-volume\") pod \"payload-processing-external-model-68467d9546-nshjz\" (UID: \"ab8d9910-c9f8-4b8b-a434-c2355fe39209\") " pod="models-as-a-service/payload-processing-external-model-68467d9546-nshjz" Nov 02 23:53:57.963227 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:57.963200 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab8d9910-c9f8-4b8b-a434-c2355fe39209-provider-credentials\") pod \"payload-processing-external-model-68467d9546-nshjz\" (UID: \"ab8d9910-c9f8-4b8b-a434-c2355fe39209\") " pod="models-as-a-service/payload-processing-external-model-68467d9546-nshjz" Nov 02 23:53:58.026558 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:58.025648 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td9fs\" (UniqueName: \"kubernetes.io/projected/ab8d9910-c9f8-4b8b-a434-c2355fe39209-kube-api-access-td9fs\") pod \"payload-processing-external-model-68467d9546-nshjz\" (UID: \"ab8d9910-c9f8-4b8b-a434-c2355fe39209\") " pod="models-as-a-service/payload-processing-external-model-68467d9546-nshjz" Nov 02 23:53:58.048943 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:58.048901 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68467d9546-nshjz" Nov 02 23:53:58.435158 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:58.434974 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-68467d9546-nshjz"] Nov 02 23:53:58.438152 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:53:58.438127 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab8d9910_c9f8_4b8b_a434_c2355fe39209.slice/crio-91d846747a73ad93de9fa2a782072e4d907546fc93a20093c88b65e190d74cfb WatchSource:0}: Error finding container 91d846747a73ad93de9fa2a782072e4d907546fc93a20093c88b65e190d74cfb: Status 404 returned error can't find the container with id 91d846747a73ad93de9fa2a782072e4d907546fc93a20093c88b65e190d74cfb Nov 02 23:53:58.800567 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:58.800532 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r"] Nov 02 23:53:58.801093 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:58.801029 5114 cpu_manager.go:414] "containerMap: removing container" podUID="d0400368-a485-486f-b433-3c456aaddff9" containerName="istio-proxy" Nov 02 23:53:58.801093 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:58.801045 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0400368-a485-486f-b433-3c456aaddff9" containerName="istio-proxy" Nov 02 23:53:58.804311 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:58.804285 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:58.862032 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:58.859876 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-98x6w\"" Nov 02 23:53:58.912037 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:58.911993 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r"] Nov 02 23:53:58.972352 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:58.972318 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3fec3e-ad31-4633-b65f-6c97883dea00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:58.972550 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:58.972363 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:58.972550 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:58.972528 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:58.972695 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:58.972594 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ljz4\" (UniqueName: \"kubernetes.io/projected/ee3fec3e-ad31-4633-b65f-6c97883dea00-kube-api-access-6ljz4\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:58.972695 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:58.972647 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:58.972765 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:58.972712 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:58.972797 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:58.972771 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:58.972849 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:58.972802 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:58.972937 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:58.972922 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:59.074671 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.074572 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3fec3e-ad31-4633-b65f-6c97883dea00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:59.074671 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.074621 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:59.074671 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.074675 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:59.074973 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.074714 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ljz4\" (UniqueName: \"kubernetes.io/projected/ee3fec3e-ad31-4633-b65f-6c97883dea00-kube-api-access-6ljz4\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:59.074973 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.074745 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:59.074973 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.074789 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:59.074973 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.074860 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:59.074973 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.074894 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:59.074973 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.074960 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:59.075296 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.075134 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:59.075296 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.075241 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:59.075440 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.075402 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:59.075440 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.075428 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:59.075659 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.075475 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3fec3e-ad31-4633-b65f-6c97883dea00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:59.077734 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.077708 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:59.077877 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.077808 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:59.135805 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.135761 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:59.136005 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.135911 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ljz4\" (UniqueName: \"kubernetes.io/projected/ee3fec3e-ad31-4633-b65f-6c97883dea00-kube-api-access-6ljz4\") pod \"maas-default-gateway-openshift-default-694db55c87-tp42r\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:59.147738 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.147693 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:53:59.361463 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.361355 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68467d9546-nshjz" event={"ID":"ab8d9910-c9f8-4b8b-a434-c2355fe39209","Type":"ContainerStarted","Data":"ce00252ac5cf29c4816ebece1a04fd8e714bdbbbe5ca7eb51d1353cdd99e4693"} Nov 02 23:53:59.361677 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.361540 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68467d9546-nshjz" event={"ID":"ab8d9910-c9f8-4b8b-a434-c2355fe39209","Type":"ContainerStarted","Data":"91d846747a73ad93de9fa2a782072e4d907546fc93a20093c88b65e190d74cfb"} Nov 02 23:53:59.361727 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.361689 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-68467d9546-nshjz" Nov 02 23:53:59.480366 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:53:59.480319 5114 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-58c6f06e6850: secret "agc-cred-58c6f06e6850" not found Nov 02 23:53:59.480366 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:53:59.480362 5114 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-68467d9546-nshjz: secret "agc-cred-58c6f06e6850" not found Nov 02 23:53:59.480615 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:53:59.480453 5114 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab8d9910-c9f8-4b8b-a434-c2355fe39209-provider-credentials podName:ab8d9910-c9f8-4b8b-a434-c2355fe39209 nodeName:}" failed. No retries permitted until 2026-11-02 23:53:59.980429073 +0000 UTC m=+3915.333462444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ab8d9910-c9f8-4b8b-a434-c2355fe39209-provider-credentials") pod "payload-processing-external-model-68467d9546-nshjz" (UID: "ab8d9910-c9f8-4b8b-a434-c2355fe39209") : secret "agc-cred-58c6f06e6850" not found Nov 02 23:53:59.493914 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.493856 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-68467d9546-nshjz" podStartSLOduration=2.493840794 podStartE2EDuration="2.493840794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:53:59.492333473 +0000 UTC m=+3914.845366850" watchObservedRunningTime="2026-11-02 23:53:59.493840794 +0000 UTC m=+3914.846874188" Nov 02 23:53:59.652056 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.652028 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r"] Nov 02 23:53:59.654060 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:53:59.654031 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3fec3e_ad31_4633_b65f_6c97883dea00.slice/crio-163441c5fbaefc7c246ef96685c060c1abf7baf72fd0a0a80b685287549dda98 WatchSource:0}: Error finding container 163441c5fbaefc7c246ef96685c060c1abf7baf72fd0a0a80b685287549dda98: Status 404 returned error can't find the container with id 163441c5fbaefc7c246ef96685c060c1abf7baf72fd0a0a80b685287549dda98 Nov 02 23:53:59.656946 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.656903 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:53:59.657105 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.657007 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:53:59.657105 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:53:59.657059 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:53:59.986052 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:53:59.985959 5114 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-58c6f06e6850: secret "agc-cred-58c6f06e6850" not found Nov 02 23:53:59.986052 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:53:59.985994 5114 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-68467d9546-nshjz: secret "agc-cred-58c6f06e6850" not found Nov 02 23:53:59.986599 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:53:59.986082 5114 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab8d9910-c9f8-4b8b-a434-c2355fe39209-provider-credentials podName:ab8d9910-c9f8-4b8b-a434-c2355fe39209 nodeName:}" failed. No retries permitted until 2026-11-02 23:54:00.986060539 +0000 UTC m=+3916.339093909 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ab8d9910-c9f8-4b8b-a434-c2355fe39209-provider-credentials") pod "payload-processing-external-model-68467d9546-nshjz" (UID: "ab8d9910-c9f8-4b8b-a434-c2355fe39209") : secret "agc-cred-58c6f06e6850" not found Nov 02 23:54:00.369432 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:00.369391 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" event={"ID":"ee3fec3e-ad31-4633-b65f-6c97883dea00","Type":"ContainerStarted","Data":"b84ad2e850d86383d6082fcd1ce0feea0f1268be615bb2f81a26f99a1b2c0fbf"} Nov 02 23:54:00.369653 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:00.369451 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" event={"ID":"ee3fec3e-ad31-4633-b65f-6c97883dea00","Type":"ContainerStarted","Data":"163441c5fbaefc7c246ef96685c060c1abf7baf72fd0a0a80b685287549dda98"} Nov 02 23:54:00.462358 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:00.462314 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-68467d9546-nshjz"] Nov 02 23:54:00.491064 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:00.491001 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" podStartSLOduration=2.490985589 podStartE2EDuration="2.490985589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:54:00.489982639 +0000 UTC m=+3915.843016031" watchObservedRunningTime="2026-11-02 23:54:00.490985589 +0000 UTC m=+3915.844018961" Nov 02 23:54:00.998833 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:54:00.998798 5114 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-58c6f06e6850: secret "agc-cred-58c6f06e6850" not found Nov 02 23:54:00.998833 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:54:00.998824 5114 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-68467d9546-nshjz: secret "agc-cred-58c6f06e6850" not found Nov 02 23:54:00.999311 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:54:00.998891 5114 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab8d9910-c9f8-4b8b-a434-c2355fe39209-provider-credentials podName:ab8d9910-c9f8-4b8b-a434-c2355fe39209 nodeName:}" failed. No retries permitted until 2026-11-02 23:54:02.99887514 +0000 UTC m=+3918.351908493 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ab8d9910-c9f8-4b8b-a434-c2355fe39209-provider-credentials") pod "payload-processing-external-model-68467d9546-nshjz" (UID: "ab8d9910-c9f8-4b8b-a434-c2355fe39209") : secret "agc-cred-58c6f06e6850" not found Nov 02 23:54:01.148139 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:01.148108 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:54:01.149621 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:01.149596 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tp42r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 02 23:54:01.149746 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:01.149646 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" podUID="ee3fec3e-ad31-4633-b65f-6c97883dea00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 02 23:54:02.148759 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:02.148722 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tp42r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 02 23:54:02.149358 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:02.148780 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" podUID="ee3fec3e-ad31-4633-b65f-6c97883dea00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 02 23:54:03.044970 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:54:03.044924 5114 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-58c6f06e6850: secret "agc-cred-58c6f06e6850" not found Nov 02 23:54:03.044970 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:54:03.044962 5114 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-68467d9546-nshjz: secret "agc-cred-58c6f06e6850" not found Nov 02 23:54:03.045199 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:54:03.045024 5114 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab8d9910-c9f8-4b8b-a434-c2355fe39209-provider-credentials podName:ab8d9910-c9f8-4b8b-a434-c2355fe39209 nodeName:}" failed. No retries permitted until 2026-11-02 23:54:07.045006978 +0000 UTC m=+3922.398040331 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ab8d9910-c9f8-4b8b-a434-c2355fe39209-provider-credentials") pod "payload-processing-external-model-68467d9546-nshjz" (UID: "ab8d9910-c9f8-4b8b-a434-c2355fe39209") : secret "agc-cred-58c6f06e6850" not found Nov 02 23:54:03.148533 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:03.148495 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tp42r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 02 23:54:03.148720 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:03.148566 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" podUID="ee3fec3e-ad31-4633-b65f-6c97883dea00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 02 23:54:04.148172 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:04.148135 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tp42r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 02 23:54:04.148634 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:04.148195 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" podUID="ee3fec3e-ad31-4633-b65f-6c97883dea00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 02 23:54:05.148466 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:05.148424 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tp42r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 02 23:54:05.148936 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:05.148551 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" podUID="ee3fec3e-ad31-4633-b65f-6c97883dea00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 02 23:54:05.493103 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:05.492473 5114 patch_prober.go:29] interesting pod/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": read tcp 10.134.0.2:54874->10.134.0.143:15021: read: connection reset by peer" start-of-body= Nov 02 23:54:05.493103 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:05.492553 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" podUID="9ab21771-e42c-462f-b83f-ee47948059dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": read tcp 10.134.0.2:54874->10.134.0.143:15021: read: connection reset by peer" Nov 02 23:54:06.148932 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:06.148895 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tp42r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 02 23:54:06.149410 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:06.148961 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" podUID="ee3fec3e-ad31-4633-b65f-6c97883dea00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 02 23:54:06.375524 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:06.375436 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-68467d9546-nshjz" podUID="ab8d9910-c9f8-4b8b-a434-c2355fe39209" containerName="payload-processing" containerID="cri-o://ce00252ac5cf29c4816ebece1a04fd8e714bdbbbe5ca7eb51d1353cdd99e4693" gracePeriod=25 Nov 02 23:54:06.407376 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:06.407344 5114 generic.go:356] "Generic (PLEG): container finished" podID="9ab21771-e42c-462f-b83f-ee47948059dd" containerID="defe1bd2fff79a70b520b761a6f75def4df0e187ed8b7b9b339e8c1d1e1bba1e" exitCode=0 Nov 02 23:54:06.407542 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:06.407429 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" event={"ID":"9ab21771-e42c-462f-b83f-ee47948059dd","Type":"ContainerDied","Data":"defe1bd2fff79a70b520b761a6f75def4df0e187ed8b7b9b339e8c1d1e1bba1e"} Nov 02 23:54:06.408020 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:06.408002 5114 scope.go:122] "RemoveContainer" containerID="defe1bd2fff79a70b520b761a6f75def4df0e187ed8b7b9b339e8c1d1e1bba1e" Nov 02 23:54:06.410040 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:06.410005 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:54:06.410123 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:06.410102 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:54:06.410171 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:06.410154 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:54:06.695652 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:06.695625 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68467d9546-nshjz" Nov 02 23:54:06.792834 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:06.792789 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab8d9910-c9f8-4b8b-a434-c2355fe39209-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab8d9910-c9f8-4b8b-a434-c2355fe39209-routing-overlay-volume\") pod \"ab8d9910-c9f8-4b8b-a434-c2355fe39209\" (UID: \"ab8d9910-c9f8-4b8b-a434-c2355fe39209\") " Nov 02 23:54:06.793029 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:06.792854 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab8d9910-c9f8-4b8b-a434-c2355fe39209-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab8d9910-c9f8-4b8b-a434-c2355fe39209-provider-credentials\") pod \"ab8d9910-c9f8-4b8b-a434-c2355fe39209\" (UID: \"ab8d9910-c9f8-4b8b-a434-c2355fe39209\") " Nov 02 23:54:06.793093 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:06.793027 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab8d9910-c9f8-4b8b-a434-c2355fe39209-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8d9910-c9f8-4b8b-a434-c2355fe39209-plugins-config-volume\") pod \"ab8d9910-c9f8-4b8b-a434-c2355fe39209\" (UID: \"ab8d9910-c9f8-4b8b-a434-c2355fe39209\") " Nov 02 23:54:06.793093 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:06.793084 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab8d9910-c9f8-4b8b-a434-c2355fe39209-kube-api-access-td9fs\" (UniqueName: \"kubernetes.io/projected/ab8d9910-c9f8-4b8b-a434-c2355fe39209-kube-api-access-td9fs\") pod \"ab8d9910-c9f8-4b8b-a434-c2355fe39209\" (UID: \"ab8d9910-c9f8-4b8b-a434-c2355fe39209\") " Nov 02 23:54:06.793260 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:06.793234 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab8d9910-c9f8-4b8b-a434-c2355fe39209-routing-overlay-volume" pod "ab8d9910-c9f8-4b8b-a434-c2355fe39209" (UID: "ab8d9910-c9f8-4b8b-a434-c2355fe39209"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:54:06.793751 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:06.793717 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab8d9910-c9f8-4b8b-a434-c2355fe39209-plugins-config-volume" pod "ab8d9910-c9f8-4b8b-a434-c2355fe39209" (UID: "ab8d9910-c9f8-4b8b-a434-c2355fe39209"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:54:06.793955 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:06.793906 5114 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab8d9910-c9f8-4b8b-a434-c2355fe39209-routing-overlay-volume\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:06.796611 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:06.796575 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab8d9910-c9f8-4b8b-a434-c2355fe39209-kube-api-access-td9fs" pod "ab8d9910-c9f8-4b8b-a434-c2355fe39209" (UID: "ab8d9910-c9f8-4b8b-a434-c2355fe39209"). InnerVolumeSpecName "kube-api-access-td9fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:54:06.796727 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:06.796656 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab8d9910-c9f8-4b8b-a434-c2355fe39209-provider-credentials" pod "ab8d9910-c9f8-4b8b-a434-c2355fe39209" (UID: "ab8d9910-c9f8-4b8b-a434-c2355fe39209"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:54:06.896423 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:06.896372 5114 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab8d9910-c9f8-4b8b-a434-c2355fe39209-provider-credentials\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:06.896423 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:06.896415 5114 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8d9910-c9f8-4b8b-a434-c2355fe39209-plugins-config-volume\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:06.896683 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:06.896432 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td9fs\" (UniqueName: \"kubernetes.io/projected/ab8d9910-c9f8-4b8b-a434-c2355fe39209-kube-api-access-td9fs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:07.151508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:07.149001 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tp42r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 02 23:54:07.151508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:07.149065 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" podUID="ee3fec3e-ad31-4633-b65f-6c97883dea00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 02 23:54:07.440046 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:07.439944 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" event={"ID":"9ab21771-e42c-462f-b83f-ee47948059dd","Type":"ContainerStarted","Data":"7bdac0b4eeae75b594b4e076c78810016f127ef6de8b16b3b0a4e886896e4117"} Nov 02 23:54:07.441654 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:07.441620 5114 generic.go:356] "Generic (PLEG): container finished" podID="ab8d9910-c9f8-4b8b-a434-c2355fe39209" containerID="ce00252ac5cf29c4816ebece1a04fd8e714bdbbbe5ca7eb51d1353cdd99e4693" exitCode=0 Nov 02 23:54:07.441831 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:07.441737 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68467d9546-nshjz" Nov 02 23:54:07.441831 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:07.441749 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68467d9546-nshjz" event={"ID":"ab8d9910-c9f8-4b8b-a434-c2355fe39209","Type":"ContainerDied","Data":"ce00252ac5cf29c4816ebece1a04fd8e714bdbbbe5ca7eb51d1353cdd99e4693"} Nov 02 23:54:07.441831 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:07.441796 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68467d9546-nshjz" event={"ID":"ab8d9910-c9f8-4b8b-a434-c2355fe39209","Type":"ContainerDied","Data":"91d846747a73ad93de9fa2a782072e4d907546fc93a20093c88b65e190d74cfb"} Nov 02 23:54:07.441831 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:07.441819 5114 scope.go:122] "RemoveContainer" containerID="ce00252ac5cf29c4816ebece1a04fd8e714bdbbbe5ca7eb51d1353cdd99e4693" Nov 02 23:54:07.452464 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:07.452433 5114 scope.go:122] "RemoveContainer" containerID="ce00252ac5cf29c4816ebece1a04fd8e714bdbbbe5ca7eb51d1353cdd99e4693" Nov 02 23:54:07.452635 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:07.452585 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:54:07.452854 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:54:07.452826 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce00252ac5cf29c4816ebece1a04fd8e714bdbbbe5ca7eb51d1353cdd99e4693\": container with ID starting with ce00252ac5cf29c4816ebece1a04fd8e714bdbbbe5ca7eb51d1353cdd99e4693 not found: ID does not exist" containerID="ce00252ac5cf29c4816ebece1a04fd8e714bdbbbe5ca7eb51d1353cdd99e4693" Nov 02 23:54:07.452930 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:07.452861 5114 patch_prober.go:29] interesting pod/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 02 23:54:07.452930 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:07.452904 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" podUID="9ab21771-e42c-462f-b83f-ee47948059dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 02 23:54:07.453070 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:07.452863 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce00252ac5cf29c4816ebece1a04fd8e714bdbbbe5ca7eb51d1353cdd99e4693"} err="failed to get container status \"ce00252ac5cf29c4816ebece1a04fd8e714bdbbbe5ca7eb51d1353cdd99e4693\": rpc error: code = NotFound desc = could not find container \"ce00252ac5cf29c4816ebece1a04fd8e714bdbbbe5ca7eb51d1353cdd99e4693\": container with ID starting with ce00252ac5cf29c4816ebece1a04fd8e714bdbbbe5ca7eb51d1353cdd99e4693 not found: ID does not exist" Nov 02 23:54:07.586105 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:54:07.586062 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:54:07.685844 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:07.685811 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-68467d9546-nshjz"] Nov 02 23:54:07.689326 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:07.689294 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-68467d9546-nshjz"] Nov 02 23:54:08.453121 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:08.453027 5114 patch_prober.go:29] interesting pod/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 02 23:54:08.453121 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:08.453096 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" podUID="9ab21771-e42c-462f-b83f-ee47948059dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 02 23:54:08.797096 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:08.797053 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab8d9910-c9f8-4b8b-a434-c2355fe39209" path="/var/lib/kubelet/pods/ab8d9910-c9f8-4b8b-a434-c2355fe39209/volumes" Nov 02 23:54:08.970044 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:08.970005 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:54:08.970556 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:08.970525 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:54:08.971541 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:08.971476 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 02 23:54:09.453849 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:09.453816 5114 patch_prober.go:29] interesting pod/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 02 23:54:09.454375 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:09.453872 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" podUID="9ab21771-e42c-462f-b83f-ee47948059dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 02 23:54:10.453421 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:10.453385 5114 patch_prober.go:29] interesting pod/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 02 23:54:10.453770 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:10.453737 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" podUID="9ab21771-e42c-462f-b83f-ee47948059dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 02 23:54:11.453681 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:11.453638 5114 patch_prober.go:29] interesting pod/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 02 23:54:11.454295 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:11.454266 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" podUID="9ab21771-e42c-462f-b83f-ee47948059dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 02 23:54:12.453696 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:12.453661 5114 patch_prober.go:29] interesting pod/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 02 23:54:12.454348 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:12.454315 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" podUID="9ab21771-e42c-462f-b83f-ee47948059dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 02 23:54:13.457889 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:13.457858 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:54:13.458372 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:13.458159 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:54:13.459086 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:13.459067 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:54:37.585171 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:54:37.585130 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:54:40.181615 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:40.181575 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7705-90505b"] Nov 02 23:54:40.182276 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:40.182224 5114 cpu_manager.go:414] "containerMap: removing container" podUID="ab8d9910-c9f8-4b8b-a434-c2355fe39209" containerName="payload-processing" Nov 02 23:54:40.182276 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:40.182247 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab8d9910-c9f8-4b8b-a434-c2355fe39209" containerName="payload-processing" Nov 02 23:54:40.182400 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:40.182355 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab8d9910-c9f8-4b8b-a434-c2355fe39209" containerName="payload-processing" Nov 02 23:54:40.184368 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:40.184348 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-90505b" Nov 02 23:54:40.239883 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:40.239839 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-90505b"] Nov 02 23:54:40.246687 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:40.246650 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqbqg\" (UniqueName: \"kubernetes.io/projected/74894f48-8a0f-487b-a299-2bfe5a4f7a17-kube-api-access-tqbqg\") pod \"test-curl-7705-90505b\" (UID: \"74894f48-8a0f-487b-a299-2bfe5a4f7a17\") " pod="opendatahub/test-curl-7705-90505b" Nov 02 23:54:40.286447 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:40.286411 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-b8v4g\"" Nov 02 23:54:40.348229 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:40.348190 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqbqg\" (UniqueName: \"kubernetes.io/projected/74894f48-8a0f-487b-a299-2bfe5a4f7a17-kube-api-access-tqbqg\") pod \"test-curl-7705-90505b\" (UID: \"74894f48-8a0f-487b-a299-2bfe5a4f7a17\") " pod="opendatahub/test-curl-7705-90505b" Nov 02 23:54:40.409452 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:40.409414 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqbqg\" (UniqueName: \"kubernetes.io/projected/74894f48-8a0f-487b-a299-2bfe5a4f7a17-kube-api-access-tqbqg\") pod \"test-curl-7705-90505b\" (UID: \"74894f48-8a0f-487b-a299-2bfe5a4f7a17\") " pod="opendatahub/test-curl-7705-90505b" Nov 02 23:54:40.495866 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:40.495764 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-90505b" Nov 02 23:54:40.869921 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:40.869896 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-90505b"] Nov 02 23:54:40.872078 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:54:40.872050 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74894f48_8a0f_487b_a299_2bfe5a4f7a17.slice/crio-c2699c2a33d5a4637e349ee18da0ac53e19494f90502011e41b41fdc6cf0b304 WatchSource:0}: Error finding container c2699c2a33d5a4637e349ee18da0ac53e19494f90502011e41b41fdc6cf0b304: Status 404 returned error can't find the container with id c2699c2a33d5a4637e349ee18da0ac53e19494f90502011e41b41fdc6cf0b304 Nov 02 23:54:41.680901 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:41.680861 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-90505b" event={"ID":"74894f48-8a0f-487b-a299-2bfe5a4f7a17","Type":"ContainerStarted","Data":"f9a033bb2f19785a017e205e4b5d001a272b4caee9ade1f240575863c7caa1c9"} Nov 02 23:54:41.681398 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:41.680916 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-90505b" event={"ID":"74894f48-8a0f-487b-a299-2bfe5a4f7a17","Type":"ContainerStarted","Data":"c2699c2a33d5a4637e349ee18da0ac53e19494f90502011e41b41fdc6cf0b304"} Nov 02 23:54:41.794297 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:41.794236 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7705-90505b" podStartSLOduration=1.794220402 podStartE2EDuration="1.794220402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:54:41.793248339 +0000 UTC m=+3957.146281714" watchObservedRunningTime="2026-11-02 23:54:41.794220402 +0000 UTC m=+3957.147253756" Nov 02 23:54:42.332047 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:42.332011 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp"] Nov 02 23:54:42.332300 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:42.332272 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" podUID="5ca490fd-a101-4a30-9bce-ca42fbe5e92b" containerName="istio-proxy" containerID="cri-o://3b1c27791388cbfa892ab8598025648924b67bdddd74f2336f8c38f339afc6a6" gracePeriod=30 Nov 02 23:54:43.301414 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:43.301376 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7705-90505b"] Nov 02 23:54:43.305819 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:43.305782 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7705-90505b"] Nov 02 23:54:43.572891 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:43.572801 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9946cb57-82svt"] Nov 02 23:54:43.573267 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:43.573252 5114 cpu_manager.go:414] "containerMap: removing container" podUID="74894f48-8a0f-487b-a299-2bfe5a4f7a17" containerName="test-curl-7705-90505b" Nov 02 23:54:43.573312 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:43.573269 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74894f48-8a0f-487b-a299-2bfe5a4f7a17" containerName="test-curl-7705-90505b" Nov 02 23:54:43.573358 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:43.573349 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="74894f48-8a0f-487b-a299-2bfe5a4f7a17" containerName="test-curl-7705-90505b" Nov 02 23:54:43.575581 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:43.575563 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9946cb57-82svt" Nov 02 23:54:43.627539 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:43.627460 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:54:43.676925 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:43.676882 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:54:43.680685 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:43.680654 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9946cb57-82svt"] Nov 02 23:54:43.688379 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:43.688345 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76db2b58-20d0-4538-9ef4-347e8a7f4301-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9946cb57-82svt\" (UID: \"76db2b58-20d0-4538-9ef4-347e8a7f4301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9946cb57-82svt" Nov 02 23:54:43.688708 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:43.688683 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jqr4\" (UniqueName: \"kubernetes.io/projected/76db2b58-20d0-4538-9ef4-347e8a7f4301-kube-api-access-6jqr4\") pod \"maas-api-revoke-sub-9946cb57-82svt\" (UID: \"76db2b58-20d0-4538-9ef4-347e8a7f4301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9946cb57-82svt" Nov 02 23:54:43.689149 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:43.689032 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76db2b58-20d0-4538-9ef4-347e8a7f4301-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9946cb57-82svt\" (UID: \"76db2b58-20d0-4538-9ef4-347e8a7f4301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9946cb57-82svt" Nov 02 23:54:43.696034 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:43.695984 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7705-90505b" podUID="74894f48-8a0f-487b-a299-2bfe5a4f7a17" containerName="test-curl-7705-90505b" containerID="cri-o://f9a033bb2f19785a017e205e4b5d001a272b4caee9ade1f240575863c7caa1c9" gracePeriod=2 Nov 02 23:54:43.761921 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:54:43.756814 5114 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7705-90505b\" is forbidden: User \"system:node:ip-10-0-135-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-250.us-west-1.compute.internal' and this object" podUID="74894f48-8a0f-487b-a299-2bfe5a4f7a17" pod="opendatahub/test-curl-7705-90505b" Nov 02 23:54:43.790143 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:43.790101 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76db2b58-20d0-4538-9ef4-347e8a7f4301-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9946cb57-82svt\" (UID: \"76db2b58-20d0-4538-9ef4-347e8a7f4301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9946cb57-82svt" Nov 02 23:54:43.790143 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:43.790153 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jqr4\" (UniqueName: \"kubernetes.io/projected/76db2b58-20d0-4538-9ef4-347e8a7f4301-kube-api-access-6jqr4\") pod \"maas-api-revoke-sub-9946cb57-82svt\" (UID: \"76db2b58-20d0-4538-9ef4-347e8a7f4301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9946cb57-82svt" Nov 02 23:54:43.790416 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:43.790305 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76db2b58-20d0-4538-9ef4-347e8a7f4301-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9946cb57-82svt\" (UID: \"76db2b58-20d0-4538-9ef4-347e8a7f4301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9946cb57-82svt" Nov 02 23:54:43.791001 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:43.790970 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76db2b58-20d0-4538-9ef4-347e8a7f4301-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9946cb57-82svt\" (UID: \"76db2b58-20d0-4538-9ef4-347e8a7f4301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9946cb57-82svt" Nov 02 23:54:43.793059 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:43.793027 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76db2b58-20d0-4538-9ef4-347e8a7f4301-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9946cb57-82svt\" (UID: \"76db2b58-20d0-4538-9ef4-347e8a7f4301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9946cb57-82svt" Nov 02 23:54:43.851282 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:43.851191 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jqr4\" (UniqueName: \"kubernetes.io/projected/76db2b58-20d0-4538-9ef4-347e8a7f4301-kube-api-access-6jqr4\") pod \"maas-api-revoke-sub-9946cb57-82svt\" (UID: \"76db2b58-20d0-4538-9ef4-347e8a7f4301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9946cb57-82svt" Nov 02 23:54:43.890189 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:43.890156 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9946cb57-82svt" Nov 02 23:54:44.139042 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.138935 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v"] Nov 02 23:54:44.142387 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.142350 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.193255 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:54:44.193211 5114 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7705-90505b\" is forbidden: User \"system:node:ip-10-0-135-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-250.us-west-1.compute.internal' and this object" podUID="74894f48-8a0f-487b-a299-2bfe5a4f7a17" pod="opendatahub/test-curl-7705-90505b" Nov 02 23:54:44.195879 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.195839 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.195965 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.195906 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.196013 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.195976 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.196071 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.196040 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.196115 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.196091 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.196226 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.196201 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9mbq\" (UniqueName: \"kubernetes.io/projected/196e5e10-3ef2-4ade-bd72-042921c2f660-kube-api-access-q9mbq\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.196413 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.196391 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.196494 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.196464 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.196540 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.196527 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/196e5e10-3ef2-4ade-bd72-042921c2f660-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.203996 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.203963 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v"] Nov 02 23:54:44.298318 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.298274 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.298550 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.298363 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.298550 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.298402 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.298550 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.298535 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.298761 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.298581 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.298761 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.298638 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9mbq\" (UniqueName: \"kubernetes.io/projected/196e5e10-3ef2-4ade-bd72-042921c2f660-kube-api-access-q9mbq\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.298761 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.298751 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.298938 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.298804 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.298938 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.298829 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/196e5e10-3ef2-4ade-bd72-042921c2f660-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.299379 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.299343 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.299855 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.299797 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.299855 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.299735 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/196e5e10-3ef2-4ade-bd72-042921c2f660-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.300034 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.299886 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.300344 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.300317 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.318039 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.301823 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.318039 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.316018 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9946cb57-82svt"] Nov 02 23:54:44.318039 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.317125 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.326541 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:54:44.326509 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76db2b58_20d0_4538_9ef4_347e8a7f4301.slice/crio-b45ee965a399e7f44c9cdd5d5676d353fe3117880df76819a907819fec91cb74 WatchSource:0}: Error finding container b45ee965a399e7f44c9cdd5d5676d353fe3117880df76819a907819fec91cb74: Status 404 returned error can't find the container with id b45ee965a399e7f44c9cdd5d5676d353fe3117880df76819a907819fec91cb74 Nov 02 23:54:44.377464 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.377417 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.377666 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.377514 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9mbq\" (UniqueName: \"kubernetes.io/projected/196e5e10-3ef2-4ade-bd72-042921c2f660-kube-api-access-q9mbq\") pod \"maas-default-gateway-openshift-default-694db55c87-fr99v\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.457831 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.457728 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:44.706673 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.706632 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9946cb57-82svt" event={"ID":"76db2b58-20d0-4538-9ef4-347e8a7f4301","Type":"ContainerStarted","Data":"553b9bd4f0b2120a0ebb836c1edfecdeac0cad05475b9e348d57c3c2e246a1a5"} Nov 02 23:54:44.706861 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.706700 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9946cb57-82svt" event={"ID":"76db2b58-20d0-4538-9ef4-347e8a7f4301","Type":"ContainerStarted","Data":"b45ee965a399e7f44c9cdd5d5676d353fe3117880df76819a907819fec91cb74"} Nov 02 23:54:44.758040 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:54:44.757943 5114 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7705-90505b\" is forbidden: User \"system:node:ip-10-0-135-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-250.us-west-1.compute.internal' and this object" podUID="74894f48-8a0f-487b-a299-2bfe5a4f7a17" pod="opendatahub/test-curl-7705-90505b" Nov 02 23:54:44.888248 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.888131 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9946cb57-82svt" podStartSLOduration=1.88811365 podStartE2EDuration="1.88811365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:54:44.877236867 +0000 UTC m=+3960.230270245" watchObservedRunningTime="2026-11-02 23:54:44.88811365 +0000 UTC m=+3960.241147025" Nov 02 23:54:44.889524 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.889472 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v"] Nov 02 23:54:44.894274 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.894228 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:54:44.894406 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.894331 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:54:44.894406 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:44.894382 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:54:45.727330 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:45.726948 5114 generic.go:356] "Generic (PLEG): container finished" podID="76db2b58-20d0-4538-9ef4-347e8a7f4301" containerID="553b9bd4f0b2120a0ebb836c1edfecdeac0cad05475b9e348d57c3c2e246a1a5" exitCode=0 Nov 02 23:54:45.727330 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:45.727069 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9946cb57-82svt" event={"ID":"76db2b58-20d0-4538-9ef4-347e8a7f4301","Type":"ContainerDied","Data":"553b9bd4f0b2120a0ebb836c1edfecdeac0cad05475b9e348d57c3c2e246a1a5"} Nov 02 23:54:45.736579 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:45.729716 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" event={"ID":"196e5e10-3ef2-4ade-bd72-042921c2f660","Type":"ContainerStarted","Data":"0b8f999a21cf11a85b0ae39f1c0667c75bc787a3b9715b65cc4d06ae7c503b1a"} Nov 02 23:54:45.736579 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:45.729784 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" event={"ID":"196e5e10-3ef2-4ade-bd72-042921c2f660","Type":"ContainerStarted","Data":"1d6510a6288d7b08f4df559b084702fa8e97eba4b48c01a6407091b182053e4d"} Nov 02 23:54:45.967917 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:45.967851 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" podStartSLOduration=1.9678285949999998 podStartE2EDuration="1.967828595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:54:45.967606586 +0000 UTC m=+3961.320639977" watchObservedRunningTime="2026-11-02 23:54:45.967828595 +0000 UTC m=+3961.320861971" Nov 02 23:54:45.979756 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:45.979729 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-90505b" Nov 02 23:54:46.030806 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:54:46.030767 5114 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7705-90505b\" is forbidden: User \"system:node:ip-10-0-135-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-250.us-west-1.compute.internal' and this object" podUID="74894f48-8a0f-487b-a299-2bfe5a4f7a17" pod="opendatahub/test-curl-7705-90505b" Nov 02 23:54:46.032145 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:46.032121 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74894f48-8a0f-487b-a299-2bfe5a4f7a17-kube-api-access-tqbqg\" (UniqueName: \"kubernetes.io/projected/74894f48-8a0f-487b-a299-2bfe5a4f7a17-kube-api-access-tqbqg\") pod \"74894f48-8a0f-487b-a299-2bfe5a4f7a17\" (UID: \"74894f48-8a0f-487b-a299-2bfe5a4f7a17\") " Nov 02 23:54:46.034508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:46.034463 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74894f48-8a0f-487b-a299-2bfe5a4f7a17-kube-api-access-tqbqg" pod "74894f48-8a0f-487b-a299-2bfe5a4f7a17" (UID: "74894f48-8a0f-487b-a299-2bfe5a4f7a17"). InnerVolumeSpecName "kube-api-access-tqbqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:54:46.133720 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:46.133680 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqbqg\" (UniqueName: \"kubernetes.io/projected/74894f48-8a0f-487b-a299-2bfe5a4f7a17-kube-api-access-tqbqg\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:46.458632 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:46.458593 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:46.460287 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:46.460248 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-fr99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 02 23:54:46.460427 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:46.460316 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" podUID="196e5e10-3ef2-4ade-bd72-042921c2f660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 02 23:54:46.737129 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:46.737034 5114 generic.go:356] "Generic (PLEG): container finished" podID="74894f48-8a0f-487b-a299-2bfe5a4f7a17" containerID="f9a033bb2f19785a017e205e4b5d001a272b4caee9ade1f240575863c7caa1c9" exitCode=137 Nov 02 23:54:46.737129 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:46.737079 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-90505b" Nov 02 23:54:46.737623 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:46.737135 5114 scope.go:122] "RemoveContainer" containerID="f9a033bb2f19785a017e205e4b5d001a272b4caee9ade1f240575863c7caa1c9" Nov 02 23:54:46.747269 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:46.747238 5114 scope.go:122] "RemoveContainer" containerID="f9a033bb2f19785a017e205e4b5d001a272b4caee9ade1f240575863c7caa1c9" Nov 02 23:54:46.747621 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:54:46.747597 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a033bb2f19785a017e205e4b5d001a272b4caee9ade1f240575863c7caa1c9\": container with ID starting with f9a033bb2f19785a017e205e4b5d001a272b4caee9ade1f240575863c7caa1c9 not found: ID does not exist" containerID="f9a033bb2f19785a017e205e4b5d001a272b4caee9ade1f240575863c7caa1c9" Nov 02 23:54:46.747696 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:46.747637 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a033bb2f19785a017e205e4b5d001a272b4caee9ade1f240575863c7caa1c9"} err="failed to get container status \"f9a033bb2f19785a017e205e4b5d001a272b4caee9ade1f240575863c7caa1c9\": rpc error: code = NotFound desc = could not find container \"f9a033bb2f19785a017e205e4b5d001a272b4caee9ade1f240575863c7caa1c9\": container with ID starting with f9a033bb2f19785a017e205e4b5d001a272b4caee9ade1f240575863c7caa1c9 not found: ID does not exist" Nov 02 23:54:46.788101 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:54:46.788063 5114 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7705-90505b\" is forbidden: User \"system:node:ip-10-0-135-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-250.us-west-1.compute.internal' and this object" podUID="74894f48-8a0f-487b-a299-2bfe5a4f7a17" pod="opendatahub/test-curl-7705-90505b" Nov 02 23:54:46.793646 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:46.793615 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74894f48-8a0f-487b-a299-2bfe5a4f7a17" path="/var/lib/kubelet/pods/74894f48-8a0f-487b-a299-2bfe5a4f7a17/volumes" Nov 02 23:54:46.840046 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:54:46.838784 5114 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7705-90505b\" is forbidden: User \"system:node:ip-10-0-135-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-250.us-west-1.compute.internal' and this object" podUID="74894f48-8a0f-487b-a299-2bfe5a4f7a17" pod="opendatahub/test-curl-7705-90505b" Nov 02 23:54:46.934977 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:46.934933 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9946cb57-82svt" Nov 02 23:54:47.044367 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.044332 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76db2b58-20d0-4538-9ef4-347e8a7f4301-kube-api-access-6jqr4\" (UniqueName: \"kubernetes.io/projected/76db2b58-20d0-4538-9ef4-347e8a7f4301-kube-api-access-6jqr4\") pod \"76db2b58-20d0-4538-9ef4-347e8a7f4301\" (UID: \"76db2b58-20d0-4538-9ef4-347e8a7f4301\") " Nov 02 23:54:47.044605 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.044399 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76db2b58-20d0-4538-9ef4-347e8a7f4301-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76db2b58-20d0-4538-9ef4-347e8a7f4301-maas-api-xks-ca\") pod \"76db2b58-20d0-4538-9ef4-347e8a7f4301\" (UID: \"76db2b58-20d0-4538-9ef4-347e8a7f4301\") " Nov 02 23:54:47.044605 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.044421 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76db2b58-20d0-4538-9ef4-347e8a7f4301-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76db2b58-20d0-4538-9ef4-347e8a7f4301-maas-api-openshift-ca\") pod \"76db2b58-20d0-4538-9ef4-347e8a7f4301\" (UID: \"76db2b58-20d0-4538-9ef4-347e8a7f4301\") " Nov 02 23:54:47.044876 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.044839 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76db2b58-20d0-4538-9ef4-347e8a7f4301-maas-api-openshift-ca" pod "76db2b58-20d0-4538-9ef4-347e8a7f4301" (UID: "76db2b58-20d0-4538-9ef4-347e8a7f4301"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:54:47.045011 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.044978 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76db2b58-20d0-4538-9ef4-347e8a7f4301-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:47.046711 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.046680 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76db2b58-20d0-4538-9ef4-347e8a7f4301-kube-api-access-6jqr4" pod "76db2b58-20d0-4538-9ef4-347e8a7f4301" (UID: "76db2b58-20d0-4538-9ef4-347e8a7f4301"). InnerVolumeSpecName "kube-api-access-6jqr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:54:47.046810 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.046705 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76db2b58-20d0-4538-9ef4-347e8a7f4301-maas-api-xks-ca" pod "76db2b58-20d0-4538-9ef4-347e8a7f4301" (UID: "76db2b58-20d0-4538-9ef4-347e8a7f4301"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:54:47.145754 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.145709 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jqr4\" (UniqueName: \"kubernetes.io/projected/76db2b58-20d0-4538-9ef4-347e8a7f4301-kube-api-access-6jqr4\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:47.145754 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.145754 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76db2b58-20d0-4538-9ef4-347e8a7f4301-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:47.459084 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.459044 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-fr99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 02 23:54:47.459256 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.459118 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" podUID="196e5e10-3ef2-4ade-bd72-042921c2f660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 02 23:54:47.647980 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.647951 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:54:47.745984 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.745952 5114 generic.go:356] "Generic (PLEG): container finished" podID="5ca490fd-a101-4a30-9bce-ca42fbe5e92b" containerID="3b1c27791388cbfa892ab8598025648924b67bdddd74f2336f8c38f339afc6a6" exitCode=0 Nov 02 23:54:47.746460 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.746005 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" Nov 02 23:54:47.746460 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.746037 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" event={"ID":"5ca490fd-a101-4a30-9bce-ca42fbe5e92b","Type":"ContainerDied","Data":"3b1c27791388cbfa892ab8598025648924b67bdddd74f2336f8c38f339afc6a6"} Nov 02 23:54:47.746460 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.746112 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp" event={"ID":"5ca490fd-a101-4a30-9bce-ca42fbe5e92b","Type":"ContainerDied","Data":"a8eb95f7e7d8e9b02b8bb7d220e0147d470d12cd9ab039e6de7e5b47ae114247"} Nov 02 23:54:47.746460 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.746146 5114 scope.go:122] "RemoveContainer" containerID="3b1c27791388cbfa892ab8598025648924b67bdddd74f2336f8c38f339afc6a6" Nov 02 23:54:47.748571 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.748550 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9946cb57-82svt" Nov 02 23:54:47.748571 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.748544 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9946cb57-82svt" event={"ID":"76db2b58-20d0-4538-9ef4-347e8a7f4301","Type":"ContainerDied","Data":"b45ee965a399e7f44c9cdd5d5676d353fe3117880df76819a907819fec91cb74"} Nov 02 23:54:47.748720 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.748605 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b45ee965a399e7f44c9cdd5d5676d353fe3117880df76819a907819fec91cb74" Nov 02 23:54:47.752385 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.752365 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-kube-api-access-4wv7l\" (UniqueName: \"kubernetes.io/projected/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-kube-api-access-4wv7l\") pod \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " Nov 02 23:54:47.752534 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.752398 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istiod-ca-cert\") pod \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " Nov 02 23:54:47.752534 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.752428 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-credential-socket\") pod \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " Nov 02 23:54:47.752534 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.752465 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-token\" (UniqueName: \"kubernetes.io/projected/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-token\") pod \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " Nov 02 23:54:47.752715 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.752550 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-data\") pod \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " Nov 02 23:54:47.752715 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.752620 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-workload-socket\") pod \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " Nov 02 23:54:47.752715 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.752660 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-podinfo\") pod \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " Nov 02 23:54:47.752715 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.752698 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-envoy\") pod \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " Nov 02 23:54:47.752935 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.752733 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-workload-certs\") pod \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\" (UID: \"5ca490fd-a101-4a30-9bce-ca42fbe5e92b\") " Nov 02 23:54:47.752935 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.752762 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-credential-socket" pod "5ca490fd-a101-4a30-9bce-ca42fbe5e92b" (UID: "5ca490fd-a101-4a30-9bce-ca42fbe5e92b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:54:47.753216 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.753191 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:47.753448 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.753385 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-data" pod "5ca490fd-a101-4a30-9bce-ca42fbe5e92b" (UID: "5ca490fd-a101-4a30-9bce-ca42fbe5e92b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:54:47.753448 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.753399 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-workload-certs" pod "5ca490fd-a101-4a30-9bce-ca42fbe5e92b" (UID: "5ca490fd-a101-4a30-9bce-ca42fbe5e92b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:54:47.753448 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.753314 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-workload-socket" pod "5ca490fd-a101-4a30-9bce-ca42fbe5e92b" (UID: "5ca490fd-a101-4a30-9bce-ca42fbe5e92b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:54:47.753789 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.753719 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istiod-ca-cert" pod "5ca490fd-a101-4a30-9bce-ca42fbe5e92b" (UID: "5ca490fd-a101-4a30-9bce-ca42fbe5e92b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:54:47.755147 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.755119 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-envoy" pod "5ca490fd-a101-4a30-9bce-ca42fbe5e92b" (UID: "5ca490fd-a101-4a30-9bce-ca42fbe5e92b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:54:47.755450 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.755413 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-kube-api-access-4wv7l" pod "5ca490fd-a101-4a30-9bce-ca42fbe5e92b" (UID: "5ca490fd-a101-4a30-9bce-ca42fbe5e92b"). InnerVolumeSpecName "kube-api-access-4wv7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:54:47.755450 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.755417 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-podinfo" pod "5ca490fd-a101-4a30-9bce-ca42fbe5e92b" (UID: "5ca490fd-a101-4a30-9bce-ca42fbe5e92b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:54:47.755615 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.755468 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-token" pod "5ca490fd-a101-4a30-9bce-ca42fbe5e92b" (UID: "5ca490fd-a101-4a30-9bce-ca42fbe5e92b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:54:47.758784 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.758758 5114 scope.go:122] "RemoveContainer" containerID="2c339568993d816a2375b9aef9a20dda76324a9adc0b7482fe0aad07098b2537" Nov 02 23:54:47.768967 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.768933 5114 scope.go:122] "RemoveContainer" containerID="3b1c27791388cbfa892ab8598025648924b67bdddd74f2336f8c38f339afc6a6" Nov 02 23:54:47.769389 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:54:47.769356 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b1c27791388cbfa892ab8598025648924b67bdddd74f2336f8c38f339afc6a6\": container with ID starting with 3b1c27791388cbfa892ab8598025648924b67bdddd74f2336f8c38f339afc6a6 not found: ID does not exist" containerID="3b1c27791388cbfa892ab8598025648924b67bdddd74f2336f8c38f339afc6a6" Nov 02 23:54:47.769529 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.769402 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b1c27791388cbfa892ab8598025648924b67bdddd74f2336f8c38f339afc6a6"} err="failed to get container status \"3b1c27791388cbfa892ab8598025648924b67bdddd74f2336f8c38f339afc6a6\": rpc error: code = NotFound desc = could not find container \"3b1c27791388cbfa892ab8598025648924b67bdddd74f2336f8c38f339afc6a6\": container with ID starting with 3b1c27791388cbfa892ab8598025648924b67bdddd74f2336f8c38f339afc6a6 not found: ID does not exist" Nov 02 23:54:47.769529 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.769429 5114 scope.go:122] "RemoveContainer" containerID="2c339568993d816a2375b9aef9a20dda76324a9adc0b7482fe0aad07098b2537" Nov 02 23:54:47.769824 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:54:47.769798 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c339568993d816a2375b9aef9a20dda76324a9adc0b7482fe0aad07098b2537\": container with ID starting with 2c339568993d816a2375b9aef9a20dda76324a9adc0b7482fe0aad07098b2537 not found: ID does not exist" containerID="2c339568993d816a2375b9aef9a20dda76324a9adc0b7482fe0aad07098b2537" Nov 02 23:54:47.769887 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.769835 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c339568993d816a2375b9aef9a20dda76324a9adc0b7482fe0aad07098b2537"} err="failed to get container status \"2c339568993d816a2375b9aef9a20dda76324a9adc0b7482fe0aad07098b2537\": rpc error: code = NotFound desc = could not find container \"2c339568993d816a2375b9aef9a20dda76324a9adc0b7482fe0aad07098b2537\": container with ID starting with 2c339568993d816a2375b9aef9a20dda76324a9adc0b7482fe0aad07098b2537 not found: ID does not exist" Nov 02 23:54:47.854555 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.854421 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:47.854555 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.854457 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:47.854555 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.854542 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:47.854555 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.854557 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:47.854824 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.854569 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:47.854824 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.854583 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:47.854824 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.854595 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wv7l\" (UniqueName: \"kubernetes.io/projected/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-kube-api-access-4wv7l\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:47.854824 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:47.854608 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ca490fd-a101-4a30-9bce-ca42fbe5e92b-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:48.196213 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:48.196126 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp"] Nov 02 23:54:48.200158 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:48.200128 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-942137-openshift-default-55dccd6d4f-rxvpp"] Nov 02 23:54:48.458800 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:48.458714 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-fr99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 02 23:54:48.458800 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:48.458777 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" podUID="196e5e10-3ef2-4ade-bd72-042921c2f660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 02 23:54:48.794363 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:48.794319 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ca490fd-a101-4a30-9bce-ca42fbe5e92b" path="/var/lib/kubelet/pods/5ca490fd-a101-4a30-9bce-ca42fbe5e92b/volumes" Nov 02 23:54:49.459218 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:49.459170 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-fr99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 02 23:54:49.459438 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:49.459234 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" podUID="196e5e10-3ef2-4ade-bd72-042921c2f660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 02 23:54:50.458380 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:50.458325 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-fr99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 02 23:54:50.458844 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:50.458402 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" podUID="196e5e10-3ef2-4ade-bd72-042921c2f660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 02 23:54:51.458987 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:51.458940 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-fr99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 02 23:54:51.459450 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:51.459015 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" podUID="196e5e10-3ef2-4ade-bd72-042921c2f660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 02 23:54:52.146508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.133350 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r"] Nov 02 23:54:52.146508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.134083 5114 cpu_manager.go:414] "containerMap: removing container" podUID="76db2b58-20d0-4538-9ef4-347e8a7f4301" containerName="revoke-keys" Nov 02 23:54:52.146508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.134102 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76db2b58-20d0-4538-9ef4-347e8a7f4301" containerName="revoke-keys" Nov 02 23:54:52.146508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.134146 5114 cpu_manager.go:414] "containerMap: removing container" podUID="5ca490fd-a101-4a30-9bce-ca42fbe5e92b" containerName="istio-proxy" Nov 02 23:54:52.146508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.134156 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ca490fd-a101-4a30-9bce-ca42fbe5e92b" containerName="istio-proxy" Nov 02 23:54:52.146508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.134171 5114 cpu_manager.go:414] "containerMap: removing container" podUID="5ca490fd-a101-4a30-9bce-ca42fbe5e92b" containerName="istio-proxy" Nov 02 23:54:52.146508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.134179 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ca490fd-a101-4a30-9bce-ca42fbe5e92b" containerName="istio-proxy" Nov 02 23:54:52.146508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.134281 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ca490fd-a101-4a30-9bce-ca42fbe5e92b" containerName="istio-proxy" Nov 02 23:54:52.146508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.134295 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ca490fd-a101-4a30-9bce-ca42fbe5e92b" containerName="istio-proxy" Nov 02 23:54:52.146508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.134309 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="76db2b58-20d0-4538-9ef4-347e8a7f4301" containerName="revoke-keys" Nov 02 23:54:52.146508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.137780 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r" Nov 02 23:54:52.194585 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.194547 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r"] Nov 02 23:54:52.212319 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.212267 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5qrm\" (UniqueName: \"kubernetes.io/projected/82af27ef-235f-4b77-bc32-69374be6e05e-kube-api-access-f5qrm\") pod \"payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r\" (UID: \"82af27ef-235f-4b77-bc32-69374be6e05e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r" Nov 02 23:54:52.212534 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.212421 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82af27ef-235f-4b77-bc32-69374be6e05e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r\" (UID: \"82af27ef-235f-4b77-bc32-69374be6e05e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r" Nov 02 23:54:52.240796 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.240762 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5db41e\"" Nov 02 23:54:52.240977 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.240920 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5db41e-dockercfg-qc75j\"" Nov 02 23:54:52.305518 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.305458 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j"] Nov 02 23:54:52.309292 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.309258 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j" Nov 02 23:54:52.314098 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.314042 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5qrm\" (UniqueName: \"kubernetes.io/projected/82af27ef-235f-4b77-bc32-69374be6e05e-kube-api-access-f5qrm\") pod \"payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r\" (UID: \"82af27ef-235f-4b77-bc32-69374be6e05e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r" Nov 02 23:54:52.314515 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.314469 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82af27ef-235f-4b77-bc32-69374be6e05e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r\" (UID: \"82af27ef-235f-4b77-bc32-69374be6e05e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r" Nov 02 23:54:52.315452 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.315430 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82af27ef-235f-4b77-bc32-69374be6e05e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r\" (UID: \"82af27ef-235f-4b77-bc32-69374be6e05e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r" Nov 02 23:54:52.367211 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.367163 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j"] Nov 02 23:54:52.375802 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.375762 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5qrm\" (UniqueName: \"kubernetes.io/projected/82af27ef-235f-4b77-bc32-69374be6e05e-kube-api-access-f5qrm\") pod \"payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r\" (UID: \"82af27ef-235f-4b77-bc32-69374be6e05e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r" Nov 02 23:54:52.416213 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.416104 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aaa2b45-4263-485e-a86b-2148f4454cbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j\" (UID: \"4aaa2b45-4263-485e-a86b-2148f4454cbe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j" Nov 02 23:54:52.416372 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.416289 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9trkz\" (UniqueName: \"kubernetes.io/projected/4aaa2b45-4263-485e-a86b-2148f4454cbe-kube-api-access-9trkz\") pod \"payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j\" (UID: \"4aaa2b45-4263-485e-a86b-2148f4454cbe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j" Nov 02 23:54:52.457064 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.457024 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r" Nov 02 23:54:52.458304 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.458282 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-fr99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 02 23:54:52.458384 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.458341 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" podUID="196e5e10-3ef2-4ade-bd72-042921c2f660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 02 23:54:52.519353 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.517421 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9trkz\" (UniqueName: \"kubernetes.io/projected/4aaa2b45-4263-485e-a86b-2148f4454cbe-kube-api-access-9trkz\") pod \"payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j\" (UID: \"4aaa2b45-4263-485e-a86b-2148f4454cbe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j" Nov 02 23:54:52.519353 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.518008 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aaa2b45-4263-485e-a86b-2148f4454cbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j\" (UID: \"4aaa2b45-4263-485e-a86b-2148f4454cbe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j" Nov 02 23:54:52.519353 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.518960 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aaa2b45-4263-485e-a86b-2148f4454cbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j\" (UID: \"4aaa2b45-4263-485e-a86b-2148f4454cbe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j" Nov 02 23:54:52.581093 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.581055 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9trkz\" (UniqueName: \"kubernetes.io/projected/4aaa2b45-4263-485e-a86b-2148f4454cbe-kube-api-access-9trkz\") pod \"payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j\" (UID: \"4aaa2b45-4263-485e-a86b-2148f4454cbe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j" Nov 02 23:54:52.624645 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.624598 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j" Nov 02 23:54:52.832626 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:52.832598 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r"] Nov 02 23:54:52.834576 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:54:52.834547 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82af27ef_235f_4b77_bc32_69374be6e05e.slice/crio-136deb78256e085dd0080187f2e46a638f065b2128b6ac00d5fc44f99fdac539 WatchSource:0}: Error finding container 136deb78256e085dd0080187f2e46a638f065b2128b6ac00d5fc44f99fdac539: Status 404 returned error can't find the container with id 136deb78256e085dd0080187f2e46a638f065b2128b6ac00d5fc44f99fdac539 Nov 02 23:54:53.011681 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:53.011654 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j"] Nov 02 23:54:53.013632 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:54:53.013604 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aaa2b45_4263_485e_a86b_2148f4454cbe.slice/crio-43a875735658da63b0a2a9c456598d0d7af45c0e35d88afec09aacd5152c64a9 WatchSource:0}: Error finding container 43a875735658da63b0a2a9c456598d0d7af45c0e35d88afec09aacd5152c64a9: Status 404 returned error can't find the container with id 43a875735658da63b0a2a9c456598d0d7af45c0e35d88afec09aacd5152c64a9 Nov 02 23:54:53.787327 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:53.787275 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j" event={"ID":"4aaa2b45-4263-485e-a86b-2148f4454cbe","Type":"ContainerStarted","Data":"a27394f5692b6fc3261de8ccca7279dff63d928cd3bd05fd878517e075834478"} Nov 02 23:54:53.787893 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:53.787355 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j" Nov 02 23:54:53.787893 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:53.787377 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j" event={"ID":"4aaa2b45-4263-485e-a86b-2148f4454cbe","Type":"ContainerStarted","Data":"43a875735658da63b0a2a9c456598d0d7af45c0e35d88afec09aacd5152c64a9"} Nov 02 23:54:53.789386 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:53.789350 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r" event={"ID":"82af27ef-235f-4b77-bc32-69374be6e05e","Type":"ContainerStarted","Data":"3fb2235f7dcd1a541579c906468f01c65b1ab16728c79b3c56e9f9b0af9c68d4"} Nov 02 23:54:53.789586 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:53.789417 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r" event={"ID":"82af27ef-235f-4b77-bc32-69374be6e05e","Type":"ContainerStarted","Data":"136deb78256e085dd0080187f2e46a638f065b2128b6ac00d5fc44f99fdac539"} Nov 02 23:54:53.789631 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:53.789612 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r" Nov 02 23:54:53.903126 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:53.903058 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j" podStartSLOduration=1.903036972 podStartE2EDuration="1.903036972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:54:53.901194082 +0000 UTC m=+3969.254227460" watchObservedRunningTime="2026-11-02 23:54:53.903036972 +0000 UTC m=+3969.256070349" Nov 02 23:54:53.905168 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:53.905132 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:53.905851 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:53.905742 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:53.906433 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:53.906412 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 02 23:54:54.017646 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:54.017572 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r" podStartSLOduration=2.017553128 podStartE2EDuration="2.017553128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:54:54.01696858 +0000 UTC m=+3969.370001956" watchObservedRunningTime="2026-11-02 23:54:54.017553128 +0000 UTC m=+3969.370586501" Nov 02 23:54:54.461441 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:54.461350 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8972c560-7vfvd"] Nov 02 23:54:54.465218 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:54.465195 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8972c560-7vfvd" Nov 02 23:54:54.520099 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:54.520061 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:54:54.520099 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:54.520084 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:54:54.522713 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:54.522685 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8972c560-7vfvd"] Nov 02 23:54:54.545995 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:54.545952 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4b518d0-6faf-4eaf-b824-9d37ea5870f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8972c560-7vfvd\" (UID: \"f4b518d0-6faf-4eaf-b824-9d37ea5870f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8972c560-7vfvd" Nov 02 23:54:54.546184 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:54.546157 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ntl5\" (UniqueName: \"kubernetes.io/projected/f4b518d0-6faf-4eaf-b824-9d37ea5870f4-kube-api-access-2ntl5\") pod \"maas-api-revoke-sub-8972c560-7vfvd\" (UID: \"f4b518d0-6faf-4eaf-b824-9d37ea5870f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8972c560-7vfvd" Nov 02 23:54:54.546362 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:54.546343 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4b518d0-6faf-4eaf-b824-9d37ea5870f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8972c560-7vfvd\" (UID: \"f4b518d0-6faf-4eaf-b824-9d37ea5870f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8972c560-7vfvd" Nov 02 23:54:54.648038 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:54.648003 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4b518d0-6faf-4eaf-b824-9d37ea5870f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8972c560-7vfvd\" (UID: \"f4b518d0-6faf-4eaf-b824-9d37ea5870f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8972c560-7vfvd" Nov 02 23:54:54.648218 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:54.648090 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ntl5\" (UniqueName: \"kubernetes.io/projected/f4b518d0-6faf-4eaf-b824-9d37ea5870f4-kube-api-access-2ntl5\") pod \"maas-api-revoke-sub-8972c560-7vfvd\" (UID: \"f4b518d0-6faf-4eaf-b824-9d37ea5870f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8972c560-7vfvd" Nov 02 23:54:54.648218 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:54.648182 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4b518d0-6faf-4eaf-b824-9d37ea5870f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8972c560-7vfvd\" (UID: \"f4b518d0-6faf-4eaf-b824-9d37ea5870f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8972c560-7vfvd" Nov 02 23:54:54.648700 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:54.648665 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4b518d0-6faf-4eaf-b824-9d37ea5870f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8972c560-7vfvd\" (UID: \"f4b518d0-6faf-4eaf-b824-9d37ea5870f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8972c560-7vfvd" Nov 02 23:54:54.650874 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:54.650846 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4b518d0-6faf-4eaf-b824-9d37ea5870f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8972c560-7vfvd\" (UID: \"f4b518d0-6faf-4eaf-b824-9d37ea5870f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8972c560-7vfvd" Nov 02 23:54:54.708619 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:54.708582 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ntl5\" (UniqueName: \"kubernetes.io/projected/f4b518d0-6faf-4eaf-b824-9d37ea5870f4-kube-api-access-2ntl5\") pod \"maas-api-revoke-sub-8972c560-7vfvd\" (UID: \"f4b518d0-6faf-4eaf-b824-9d37ea5870f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8972c560-7vfvd" Nov 02 23:54:54.779178 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:54.779084 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8972c560-7vfvd" Nov 02 23:54:55.160819 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:55.160788 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8972c560-7vfvd"] Nov 02 23:54:55.162917 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:54:55.162876 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b518d0_6faf_4eaf_b824_9d37ea5870f4.slice/crio-30147ebb0142e1d554c17c4910b8fb5e1c7fc3cc32165058e8811acf8b326a2b WatchSource:0}: Error finding container 30147ebb0142e1d554c17c4910b8fb5e1c7fc3cc32165058e8811acf8b326a2b: Status 404 returned error can't find the container with id 30147ebb0142e1d554c17c4910b8fb5e1c7fc3cc32165058e8811acf8b326a2b Nov 02 23:54:55.802063 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:55.802019 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8972c560-7vfvd" event={"ID":"f4b518d0-6faf-4eaf-b824-9d37ea5870f4","Type":"ContainerStarted","Data":"fb253bd8d0e3754c1b4c4628c86ca26e716844a9a9b68f99a141acd9cf11f13d"} Nov 02 23:54:55.802266 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:55.802089 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8972c560-7vfvd" event={"ID":"f4b518d0-6faf-4eaf-b824-9d37ea5870f4","Type":"ContainerStarted","Data":"30147ebb0142e1d554c17c4910b8fb5e1c7fc3cc32165058e8811acf8b326a2b"} Nov 02 23:54:55.915958 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:55.915882 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8972c560-7vfvd" podStartSLOduration=1.915863056 podStartE2EDuration="1.915863056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:54:55.915606081 +0000 UTC m=+3971.268639459" watchObservedRunningTime="2026-11-02 23:54:55.915863056 +0000 UTC m=+3971.268896434" Nov 02 23:54:56.810228 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:56.810194 5114 generic.go:356] "Generic (PLEG): container finished" podID="f4b518d0-6faf-4eaf-b824-9d37ea5870f4" containerID="fb253bd8d0e3754c1b4c4628c86ca26e716844a9a9b68f99a141acd9cf11f13d" exitCode=0 Nov 02 23:54:56.810698 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:56.810292 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8972c560-7vfvd" event={"ID":"f4b518d0-6faf-4eaf-b824-9d37ea5870f4","Type":"ContainerDied","Data":"fb253bd8d0e3754c1b4c4628c86ca26e716844a9a9b68f99a141acd9cf11f13d"} Nov 02 23:54:57.964110 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:57.964076 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8972c560-7vfvd" Nov 02 23:54:58.090134 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:58.090019 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4b518d0-6faf-4eaf-b824-9d37ea5870f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4b518d0-6faf-4eaf-b824-9d37ea5870f4-maas-api-openshift-ca\") pod \"f4b518d0-6faf-4eaf-b824-9d37ea5870f4\" (UID: \"f4b518d0-6faf-4eaf-b824-9d37ea5870f4\") " Nov 02 23:54:58.090134 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:58.090072 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4b518d0-6faf-4eaf-b824-9d37ea5870f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4b518d0-6faf-4eaf-b824-9d37ea5870f4-maas-api-xks-ca\") pod \"f4b518d0-6faf-4eaf-b824-9d37ea5870f4\" (UID: \"f4b518d0-6faf-4eaf-b824-9d37ea5870f4\") " Nov 02 23:54:58.090388 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:58.090306 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b518d0-6faf-4eaf-b824-9d37ea5870f4-kube-api-access-2ntl5\" (UniqueName: \"kubernetes.io/projected/f4b518d0-6faf-4eaf-b824-9d37ea5870f4-kube-api-access-2ntl5\") pod \"f4b518d0-6faf-4eaf-b824-9d37ea5870f4\" (UID: \"f4b518d0-6faf-4eaf-b824-9d37ea5870f4\") " Nov 02 23:54:58.090388 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:58.090351 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b518d0-6faf-4eaf-b824-9d37ea5870f4-maas-api-openshift-ca" pod "f4b518d0-6faf-4eaf-b824-9d37ea5870f4" (UID: "f4b518d0-6faf-4eaf-b824-9d37ea5870f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:54:58.090765 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:58.090742 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4b518d0-6faf-4eaf-b824-9d37ea5870f4-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:58.092585 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:58.092558 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b518d0-6faf-4eaf-b824-9d37ea5870f4-kube-api-access-2ntl5" pod "f4b518d0-6faf-4eaf-b824-9d37ea5870f4" (UID: "f4b518d0-6faf-4eaf-b824-9d37ea5870f4"). InnerVolumeSpecName "kube-api-access-2ntl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:54:58.092585 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:58.092555 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4b518d0-6faf-4eaf-b824-9d37ea5870f4-maas-api-xks-ca" pod "f4b518d0-6faf-4eaf-b824-9d37ea5870f4" (UID: "f4b518d0-6faf-4eaf-b824-9d37ea5870f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:54:58.191560 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:58.191520 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ntl5\" (UniqueName: \"kubernetes.io/projected/f4b518d0-6faf-4eaf-b824-9d37ea5870f4-kube-api-access-2ntl5\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:58.191560 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:58.191554 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4b518d0-6faf-4eaf-b824-9d37ea5870f4-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:54:58.823400 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:58.823370 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8972c560-7vfvd" Nov 02 23:54:58.823573 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:58.823401 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8972c560-7vfvd" event={"ID":"f4b518d0-6faf-4eaf-b824-9d37ea5870f4","Type":"ContainerDied","Data":"30147ebb0142e1d554c17c4910b8fb5e1c7fc3cc32165058e8811acf8b326a2b"} Nov 02 23:54:58.823573 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:54:58.823454 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30147ebb0142e1d554c17c4910b8fb5e1c7fc3cc32165058e8811acf8b326a2b" Nov 02 23:55:02.248356 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.248317 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w"] Nov 02 23:55:02.248857 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.248839 5114 cpu_manager.go:414] "containerMap: removing container" podUID="f4b518d0-6faf-4eaf-b824-9d37ea5870f4" containerName="revoke-keys" Nov 02 23:55:02.248924 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.248858 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b518d0-6faf-4eaf-b824-9d37ea5870f4" containerName="revoke-keys" Nov 02 23:55:02.248980 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.248970 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b518d0-6faf-4eaf-b824-9d37ea5870f4" containerName="revoke-keys" Nov 02 23:55:02.252093 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.252077 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.313673 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.306857 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-5e7e9e-openshift-default-dockercfg-zmpfd\"" Nov 02 23:55:02.357377 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.354190 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-podinfo\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.357377 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.354277 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-data\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.357377 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.354327 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-envoy\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.357377 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.354364 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7zwq\" (UniqueName: \"kubernetes.io/projected/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-kube-api-access-c7zwq\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.357377 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.354434 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-workload-socket\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.357377 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.354518 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-workload-certs\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.357377 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.354544 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istiod-ca-cert\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.357377 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.354623 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-credential-socket\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.357377 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.354676 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-token\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.362504 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.360854 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w"] Nov 02 23:55:02.455971 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.455929 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-workload-certs\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.455971 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.455971 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istiod-ca-cert\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.456233 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.456028 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-credential-socket\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.456233 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.456066 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-token\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.456747 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.456716 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-podinfo\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.456945 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.456922 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-workload-certs\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.458806 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.456796 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-credential-socket\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.458806 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.457467 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istiod-ca-cert\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.458806 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.457719 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-data\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.458806 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.457768 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-envoy\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.458806 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.457815 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7zwq\" (UniqueName: \"kubernetes.io/projected/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-kube-api-access-c7zwq\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.458806 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.457893 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-workload-socket\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.458806 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.458275 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-workload-socket\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.458806 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.458556 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-data\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.468677 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.468633 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-podinfo\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.469834 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.469805 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-envoy\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.517458 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.517383 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-token\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.518926 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.518902 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7zwq\" (UniqueName: \"kubernetes.io/projected/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-kube-api-access-c7zwq\") pod \"e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:02.569039 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:02.568993 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:03.003642 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:03.003611 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w"] Nov 02 23:55:03.005007 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:55:03.004976 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53ad7b3_45b4_4450_a3c7_b7e00596d28d.slice/crio-d879703a5f01086b9352e081364c237309edab24bf8060a3de5f59e6aff8db4c WatchSource:0}: Error finding container d879703a5f01086b9352e081364c237309edab24bf8060a3de5f59e6aff8db4c: Status 404 returned error can't find the container with id d879703a5f01086b9352e081364c237309edab24bf8060a3de5f59e6aff8db4c Nov 02 23:55:03.007667 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:03.007627 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:55:03.007778 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:03.007702 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:55:03.007778 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:03.007733 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:55:03.854348 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:03.854313 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" event={"ID":"a53ad7b3-45b4-4450-a3c7-b7e00596d28d","Type":"ContainerStarted","Data":"c8f3b0709d5781d8731b8bbed90346192d2c62de79b750a46ff9b1a1840f953e"} Nov 02 23:55:03.854754 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:03.854370 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" event={"ID":"a53ad7b3-45b4-4450-a3c7-b7e00596d28d","Type":"ContainerStarted","Data":"d879703a5f01086b9352e081364c237309edab24bf8060a3de5f59e6aff8db4c"} Nov 02 23:55:04.012751 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:04.010852 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" podStartSLOduration=2.010831571 podStartE2EDuration="2.010831571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:55:04.008324649 +0000 UTC m=+3979.361358039" watchObservedRunningTime="2026-11-02 23:55:04.010831571 +0000 UTC m=+3979.363864947" Nov 02 23:55:04.569594 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:04.569554 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:04.575809 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:04.575779 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:04.798433 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:04.798400 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j" Nov 02 23:55:04.798661 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:04.798512 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r" Nov 02 23:55:04.861158 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:04.861058 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:04.862149 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:04.862129 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:55:07.583901 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:55:07.583857 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:55:08.314266 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:08.314230 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl"] Nov 02 23:55:23.335586 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.335524 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" podUID="642fc4ab-f938-46bb-9ee1-eb815c9d956b" containerName="main" containerID="cri-o://d922f16cd3a1d18b95f15850d673a9b49cf9aca3e717b9d19d3e979a0a94cba7" gracePeriod=45 Nov 02 23:55:23.616530 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.616500 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:55:23.723143 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.723103 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-tmp-dir\") pod \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\" (UID: \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\") " Nov 02 23:55:23.723350 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.723158 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-home\" (UniqueName: \"kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-home\") pod \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\" (UID: \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\") " Nov 02 23:55:23.723350 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.723210 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-dshm\") pod \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\" (UID: \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\") " Nov 02 23:55:23.723350 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.723289 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/642fc4ab-f938-46bb-9ee1-eb815c9d956b-kube-api-access-npvjb\" (UniqueName: \"kubernetes.io/projected/642fc4ab-f938-46bb-9ee1-eb815c9d956b-kube-api-access-npvjb\") pod \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\" (UID: \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\") " Nov 02 23:55:23.723588 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.723380 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-model-cache\") pod \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\" (UID: \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\") " Nov 02 23:55:23.723588 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.723443 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/642fc4ab-f938-46bb-9ee1-eb815c9d956b-tls-certs\" (UniqueName: \"kubernetes.io/secret/642fc4ab-f938-46bb-9ee1-eb815c9d956b-tls-certs\") pod \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\" (UID: \"642fc4ab-f938-46bb-9ee1-eb815c9d956b\") " Nov 02 23:55:23.723588 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.723465 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-tmp-dir" pod "642fc4ab-f938-46bb-9ee1-eb815c9d956b" (UID: "642fc4ab-f938-46bb-9ee1-eb815c9d956b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:55:23.723588 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.723460 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-home" pod "642fc4ab-f938-46bb-9ee1-eb815c9d956b" (UID: "642fc4ab-f938-46bb-9ee1-eb815c9d956b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:55:23.723811 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.723773 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-model-cache" pod "642fc4ab-f938-46bb-9ee1-eb815c9d956b" (UID: "642fc4ab-f938-46bb-9ee1-eb815c9d956b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:55:23.724032 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.724013 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:55:23.724102 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.724062 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:55:23.724102 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.724078 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:55:23.725889 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.725864 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/642fc4ab-f938-46bb-9ee1-eb815c9d956b-tls-certs" pod "642fc4ab-f938-46bb-9ee1-eb815c9d956b" (UID: "642fc4ab-f938-46bb-9ee1-eb815c9d956b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:55:23.726009 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.725890 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/642fc4ab-f938-46bb-9ee1-eb815c9d956b-kube-api-access-npvjb" pod "642fc4ab-f938-46bb-9ee1-eb815c9d956b" (UID: "642fc4ab-f938-46bb-9ee1-eb815c9d956b"). InnerVolumeSpecName "kube-api-access-npvjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:55:23.726009 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.725902 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-dshm" pod "642fc4ab-f938-46bb-9ee1-eb815c9d956b" (UID: "642fc4ab-f938-46bb-9ee1-eb815c9d956b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:55:23.825941 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.825901 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/642fc4ab-f938-46bb-9ee1-eb815c9d956b-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:55:23.825941 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.825936 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npvjb\" (UniqueName: \"kubernetes.io/projected/642fc4ab-f938-46bb-9ee1-eb815c9d956b-kube-api-access-npvjb\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:55:23.825941 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.825946 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/642fc4ab-f938-46bb-9ee1-eb815c9d956b-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:55:23.980727 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.980640 5114 generic.go:356] "Generic (PLEG): container finished" podID="642fc4ab-f938-46bb-9ee1-eb815c9d956b" containerID="d922f16cd3a1d18b95f15850d673a9b49cf9aca3e717b9d19d3e979a0a94cba7" exitCode=0 Nov 02 23:55:23.980727 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.980691 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" Nov 02 23:55:23.980934 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.980730 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" event={"ID":"642fc4ab-f938-46bb-9ee1-eb815c9d956b","Type":"ContainerDied","Data":"d922f16cd3a1d18b95f15850d673a9b49cf9aca3e717b9d19d3e979a0a94cba7"} Nov 02 23:55:23.980934 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.980776 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl" event={"ID":"642fc4ab-f938-46bb-9ee1-eb815c9d956b","Type":"ContainerDied","Data":"f07962043935f24c6a05a7113ee4921073aaf5fb01ce84ba66178698ef0c7016"} Nov 02 23:55:23.980934 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.980793 5114 scope.go:122] "RemoveContainer" containerID="d922f16cd3a1d18b95f15850d673a9b49cf9aca3e717b9d19d3e979a0a94cba7" Nov 02 23:55:23.991035 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.991009 5114 scope.go:122] "RemoveContainer" containerID="d922f16cd3a1d18b95f15850d673a9b49cf9aca3e717b9d19d3e979a0a94cba7" Nov 02 23:55:23.991360 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:55:23.991335 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d922f16cd3a1d18b95f15850d673a9b49cf9aca3e717b9d19d3e979a0a94cba7\": container with ID starting with d922f16cd3a1d18b95f15850d673a9b49cf9aca3e717b9d19d3e979a0a94cba7 not found: ID does not exist" containerID="d922f16cd3a1d18b95f15850d673a9b49cf9aca3e717b9d19d3e979a0a94cba7" Nov 02 23:55:23.991417 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:23.991376 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d922f16cd3a1d18b95f15850d673a9b49cf9aca3e717b9d19d3e979a0a94cba7"} err="failed to get container status \"d922f16cd3a1d18b95f15850d673a9b49cf9aca3e717b9d19d3e979a0a94cba7\": rpc error: code = NotFound desc = could not find container \"d922f16cd3a1d18b95f15850d673a9b49cf9aca3e717b9d19d3e979a0a94cba7\": container with ID starting with d922f16cd3a1d18b95f15850d673a9b49cf9aca3e717b9d19d3e979a0a94cba7 not found: ID does not exist" Nov 02 23:55:24.145269 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:24.145219 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl"] Nov 02 23:55:24.194173 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:24.194138 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-882a2a/auth-test-model-882a2a-kserve-64f9bf596f-lbhtl"] Nov 02 23:55:24.793998 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:24.793966 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="642fc4ab-f938-46bb-9ee1-eb815c9d956b" path="/var/lib/kubelet/pods/642fc4ab-f938-46bb-9ee1-eb815c9d956b/volumes" Nov 02 23:55:29.495608 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.495566 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt"] Nov 02 23:55:29.496051 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.496037 5114 cpu_manager.go:414] "containerMap: removing container" podUID="642fc4ab-f938-46bb-9ee1-eb815c9d956b" containerName="main" Nov 02 23:55:29.496051 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.496049 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="642fc4ab-f938-46bb-9ee1-eb815c9d956b" containerName="main" Nov 02 23:55:29.496142 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.496130 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="642fc4ab-f938-46bb-9ee1-eb815c9d956b" containerName="main" Nov 02 23:55:29.501062 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.501034 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.605500 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.605439 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt"] Nov 02 23:55:29.693529 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.693463 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.693739 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.693542 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.693739 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.693643 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.693739 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.693680 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z5x7\" (UniqueName: \"kubernetes.io/projected/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-kube-api-access-9z5x7\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.693739 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.693701 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.693885 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.693818 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.693971 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.693947 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.694033 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.694015 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.694075 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.694049 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.795615 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.795565 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.795841 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.795638 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.795841 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.795674 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.795841 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.795718 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.795841 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.795774 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.795841 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.795821 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.796113 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.795846 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z5x7\" (UniqueName: \"kubernetes.io/projected/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-kube-api-access-9z5x7\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.796113 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.795871 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.796113 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.795917 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.796264 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.796107 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.796264 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.796176 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.796372 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.796306 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.796607 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.796584 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.797009 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.796746 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.798304 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.798280 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.798742 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.798723 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.857024 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.856983 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:29.857024 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:29.857014 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z5x7\" (UniqueName: \"kubernetes.io/projected/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-kube-api-access-9z5x7\") pod \"maas-default-gateway-openshift-default-694db55c87-4wgnt\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:30.114836 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:30.114749 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:30.552620 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:30.552587 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt"] Nov 02 23:55:30.553174 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:55:30.553148 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc58f9ea_5fa3_44d0_975f_1fed35f2463b.slice/crio-7e00f9c5786bc6c2598ca98c99e5d5fe73e76d96a86b29c85ed8da9b55ce552f WatchSource:0}: Error finding container 7e00f9c5786bc6c2598ca98c99e5d5fe73e76d96a86b29c85ed8da9b55ce552f: Status 404 returned error can't find the container with id 7e00f9c5786bc6c2598ca98c99e5d5fe73e76d96a86b29c85ed8da9b55ce552f Nov 02 23:55:30.556013 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:30.555982 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:55:30.556099 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:30.556050 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:55:30.556099 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:30.556081 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:55:31.020676 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:31.020570 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" event={"ID":"bc58f9ea-5fa3-44d0-975f-1fed35f2463b","Type":"ContainerStarted","Data":"0cffd097a2334d4404d9c42a08ba898c8eb2adf70e5dc45640030e365ca82cda"} Nov 02 23:55:31.020676 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:31.020639 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" event={"ID":"bc58f9ea-5fa3-44d0-975f-1fed35f2463b","Type":"ContainerStarted","Data":"7e00f9c5786bc6c2598ca98c99e5d5fe73e76d96a86b29c85ed8da9b55ce552f"} Nov 02 23:55:31.115727 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:31.115689 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:31.117281 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:31.117249 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4wgnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 23:55:31.117447 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:31.117320 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" podUID="bc58f9ea-5fa3-44d0-975f-1fed35f2463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 23:55:31.141333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:31.141252 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" podStartSLOduration=2.141229156 podStartE2EDuration="2.141229156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:55:31.140077401 +0000 UTC m=+4006.493110777" watchObservedRunningTime="2026-11-02 23:55:31.141229156 +0000 UTC m=+4006.494262532" Nov 02 23:55:32.117032 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:32.116981 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4wgnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 23:55:32.119099 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:32.119063 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" podUID="bc58f9ea-5fa3-44d0-975f-1fed35f2463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 23:55:33.116518 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:33.115865 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4wgnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 23:55:33.116518 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:33.115958 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" podUID="bc58f9ea-5fa3-44d0-975f-1fed35f2463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 23:55:34.130997 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:34.130957 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4wgnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 23:55:34.131928 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:34.131674 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" podUID="bc58f9ea-5fa3-44d0-975f-1fed35f2463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 23:55:35.135895 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:35.135855 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4wgnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 23:55:35.136752 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:35.136717 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" podUID="bc58f9ea-5fa3-44d0-975f-1fed35f2463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 23:55:36.120514 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:36.119928 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4wgnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 23:55:36.120514 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:36.119995 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" podUID="bc58f9ea-5fa3-44d0-975f-1fed35f2463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 23:55:37.115468 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:37.115431 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4wgnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 23:55:37.116071 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:37.115535 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" podUID="bc58f9ea-5fa3-44d0-975f-1fed35f2463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 23:55:37.585707 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:55:37.585667 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:55:38.115636 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:38.115598 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4wgnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 23:55:38.116149 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:38.115680 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" podUID="bc58f9ea-5fa3-44d0-975f-1fed35f2463b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 23:55:38.938432 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:38.938381 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7705-920502"] Nov 02 23:55:38.942918 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:38.942886 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-920502" Nov 02 23:55:38.996941 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:38.996905 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-920502"] Nov 02 23:55:39.032501 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:39.032440 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w79f\" (UniqueName: \"kubernetes.io/projected/9f6ddbe5-1ee7-4da0-be27-876b04ca0e2a-kube-api-access-2w79f\") pod \"test-curl-7705-920502\" (UID: \"9f6ddbe5-1ee7-4da0-be27-876b04ca0e2a\") " pod="opendatahub/test-curl-7705-920502" Nov 02 23:55:39.042055 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:39.042023 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-b8v4g\"" Nov 02 23:55:39.120614 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:39.120585 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:39.121024 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:39.120942 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:39.121644 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:39.121622 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 02 23:55:39.133261 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:39.133233 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w79f\" (UniqueName: \"kubernetes.io/projected/9f6ddbe5-1ee7-4da0-be27-876b04ca0e2a-kube-api-access-2w79f\") pod \"test-curl-7705-920502\" (UID: \"9f6ddbe5-1ee7-4da0-be27-876b04ca0e2a\") " pod="opendatahub/test-curl-7705-920502" Nov 02 23:55:39.194122 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:39.194042 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w79f\" (UniqueName: \"kubernetes.io/projected/9f6ddbe5-1ee7-4da0-be27-876b04ca0e2a-kube-api-access-2w79f\") pod \"test-curl-7705-920502\" (UID: \"9f6ddbe5-1ee7-4da0-be27-876b04ca0e2a\") " pod="opendatahub/test-curl-7705-920502" Nov 02 23:55:39.259197 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:39.259165 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-920502" Nov 02 23:55:39.642012 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:39.641950 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-920502"] Nov 02 23:55:39.644679 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:55:39.644648 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f6ddbe5_1ee7_4da0_be27_876b04ca0e2a.slice/crio-d91e1ec26ac4362b76124010de4864964da2454edf15c3fd72e649d946bcb324 WatchSource:0}: Error finding container d91e1ec26ac4362b76124010de4864964da2454edf15c3fd72e649d946bcb324: Status 404 returned error can't find the container with id d91e1ec26ac4362b76124010de4864964da2454edf15c3fd72e649d946bcb324 Nov 02 23:55:40.085758 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:40.085722 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-920502" event={"ID":"9f6ddbe5-1ee7-4da0-be27-876b04ca0e2a","Type":"ContainerStarted","Data":"208d1052cb0a371d50912e4a8a90a10ee95e273dae076df97f8d8f637d3ba202"} Nov 02 23:55:40.085945 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:40.085768 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-920502" event={"ID":"9f6ddbe5-1ee7-4da0-be27-876b04ca0e2a","Type":"ContainerStarted","Data":"d91e1ec26ac4362b76124010de4864964da2454edf15c3fd72e649d946bcb324"} Nov 02 23:55:40.199004 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:40.198937 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7705-920502" podStartSLOduration=2.198921377 podStartE2EDuration="2.198921377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:55:40.198174874 +0000 UTC m=+4015.551208250" watchObservedRunningTime="2026-11-02 23:55:40.198921377 +0000 UTC m=+4015.551954751" Nov 02 23:55:41.606078 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:41.606031 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7705-920502"] Nov 02 23:55:41.609423 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:41.609386 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7705-920502"] Nov 02 23:55:42.098351 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:42.098289 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7705-920502" podUID="9f6ddbe5-1ee7-4da0-be27-876b04ca0e2a" containerName="test-curl-7705-920502" containerID="cri-o://208d1052cb0a371d50912e4a8a90a10ee95e273dae076df97f8d8f637d3ba202" gracePeriod=2 Nov 02 23:55:42.149158 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:55:42.149112 5114 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7705-920502\" is forbidden: User \"system:node:ip-10-0-135-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-250.us-west-1.compute.internal' and this object" podUID="9f6ddbe5-1ee7-4da0-be27-876b04ca0e2a" pod="opendatahub/test-curl-7705-920502" Nov 02 23:55:44.342906 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.342873 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-920502" Nov 02 23:55:44.496414 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.496329 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f6ddbe5-1ee7-4da0-be27-876b04ca0e2a-kube-api-access-2w79f\" (UniqueName: \"kubernetes.io/projected/9f6ddbe5-1ee7-4da0-be27-876b04ca0e2a-kube-api-access-2w79f\") pod \"9f6ddbe5-1ee7-4da0-be27-876b04ca0e2a\" (UID: \"9f6ddbe5-1ee7-4da0-be27-876b04ca0e2a\") " Nov 02 23:55:44.498640 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.498617 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f6ddbe5-1ee7-4da0-be27-876b04ca0e2a-kube-api-access-2w79f" pod "9f6ddbe5-1ee7-4da0-be27-876b04ca0e2a" (UID: "9f6ddbe5-1ee7-4da0-be27-876b04ca0e2a"). InnerVolumeSpecName "kube-api-access-2w79f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:55:44.598130 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.598094 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w79f\" (UniqueName: \"kubernetes.io/projected/9f6ddbe5-1ee7-4da0-be27-876b04ca0e2a-kube-api-access-2w79f\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:55:44.632396 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.632361 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655"] Nov 02 23:55:44.632887 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.632871 5114 cpu_manager.go:414] "containerMap: removing container" podUID="9f6ddbe5-1ee7-4da0-be27-876b04ca0e2a" containerName="test-curl-7705-920502" Nov 02 23:55:44.632962 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.632889 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f6ddbe5-1ee7-4da0-be27-876b04ca0e2a" containerName="test-curl-7705-920502" Nov 02 23:55:44.633019 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.633006 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f6ddbe5-1ee7-4da0-be27-876b04ca0e2a" containerName="test-curl-7705-920502" Nov 02 23:55:44.636342 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.636322 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.698903 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.698861 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.699091 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.698921 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.699091 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.699066 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.699167 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.699124 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6g5s\" (UniqueName: \"kubernetes.io/projected/7701f578-a649-4f37-956b-aa1189848710-kube-api-access-g6g5s\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.699256 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.699238 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7701f578-a649-4f37-956b-aa1189848710-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.699333 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.699318 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7701f578-a649-4f37-956b-aa1189848710-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.699370 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.699352 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7701f578-a649-4f37-956b-aa1189848710-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.699447 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.699427 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.699546 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.699529 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.741737 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.741699 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655"] Nov 02 23:55:44.793298 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.793266 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f6ddbe5-1ee7-4da0-be27-876b04ca0e2a" path="/var/lib/kubelet/pods/9f6ddbe5-1ee7-4da0-be27-876b04ca0e2a/volumes" Nov 02 23:55:44.801737 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.801692 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.801926 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.801761 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.801926 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.801825 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.801926 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.801879 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.802095 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.801943 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.802095 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.801985 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6g5s\" (UniqueName: \"kubernetes.io/projected/7701f578-a649-4f37-956b-aa1189848710-kube-api-access-g6g5s\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.802095 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.802064 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7701f578-a649-4f37-956b-aa1189848710-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.802247 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.802124 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7701f578-a649-4f37-956b-aa1189848710-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.802247 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.802157 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7701f578-a649-4f37-956b-aa1189848710-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.803102 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.803069 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7701f578-a649-4f37-956b-aa1189848710-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.803413 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.803386 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.805002 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.804975 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.805247 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.805228 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.805334 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.805303 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.805599 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.805497 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.809750 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.809724 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7701f578-a649-4f37-956b-aa1189848710-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.871265 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.871226 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6g5s\" (UniqueName: \"kubernetes.io/projected/7701f578-a649-4f37-956b-aa1189848710-kube-api-access-g6g5s\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.871781 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.871759 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7701f578-a649-4f37-956b-aa1189848710-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-gd655\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:44.951273 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:44.951230 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:45.123836 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:45.123742 5114 generic.go:356] "Generic (PLEG): container finished" podID="9f6ddbe5-1ee7-4da0-be27-876b04ca0e2a" containerID="208d1052cb0a371d50912e4a8a90a10ee95e273dae076df97f8d8f637d3ba202" exitCode=137 Nov 02 23:55:45.123836 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:45.123812 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-920502" Nov 02 23:55:45.124071 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:45.123844 5114 scope.go:122] "RemoveContainer" containerID="208d1052cb0a371d50912e4a8a90a10ee95e273dae076df97f8d8f637d3ba202" Nov 02 23:55:45.132907 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:45.132888 5114 scope.go:122] "RemoveContainer" containerID="208d1052cb0a371d50912e4a8a90a10ee95e273dae076df97f8d8f637d3ba202" Nov 02 23:55:45.133205 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:55:45.133178 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"208d1052cb0a371d50912e4a8a90a10ee95e273dae076df97f8d8f637d3ba202\": container with ID starting with 208d1052cb0a371d50912e4a8a90a10ee95e273dae076df97f8d8f637d3ba202 not found: ID does not exist" containerID="208d1052cb0a371d50912e4a8a90a10ee95e273dae076df97f8d8f637d3ba202" Nov 02 23:55:45.133270 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:45.133207 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"208d1052cb0a371d50912e4a8a90a10ee95e273dae076df97f8d8f637d3ba202"} err="failed to get container status \"208d1052cb0a371d50912e4a8a90a10ee95e273dae076df97f8d8f637d3ba202\": rpc error: code = NotFound desc = could not find container \"208d1052cb0a371d50912e4a8a90a10ee95e273dae076df97f8d8f637d3ba202\": container with ID starting with 208d1052cb0a371d50912e4a8a90a10ee95e273dae076df97f8d8f637d3ba202 not found: ID does not exist" Nov 02 23:55:45.382202 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:45.382176 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655"] Nov 02 23:55:45.383979 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:55:45.383938 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7701f578_a649_4f37_956b_aa1189848710.slice/crio-8a6e3f2f7eb2c19bae79a758e5a436f1874bed131fcd94f53f73a049ed352616 WatchSource:0}: Error finding container 8a6e3f2f7eb2c19bae79a758e5a436f1874bed131fcd94f53f73a049ed352616: Status 404 returned error can't find the container with id 8a6e3f2f7eb2c19bae79a758e5a436f1874bed131fcd94f53f73a049ed352616 Nov 02 23:55:45.386850 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:45.386811 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:55:45.386964 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:45.386899 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:55:45.386964 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:45.386946 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:55:46.133044 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:46.133010 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" event={"ID":"7701f578-a649-4f37-956b-aa1189848710","Type":"ContainerStarted","Data":"7c76cb00678d5ebb455a7e54711c4ed9dfe626a0ff39c851bb0ac61e5190db4a"} Nov 02 23:55:46.133166 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:46.133075 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" event={"ID":"7701f578-a649-4f37-956b-aa1189848710","Type":"ContainerStarted","Data":"8a6e3f2f7eb2c19bae79a758e5a436f1874bed131fcd94f53f73a049ed352616"} Nov 02 23:55:46.253003 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:46.252931 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" podStartSLOduration=2.25291099 podStartE2EDuration="2.25291099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:55:46.252401501 +0000 UTC m=+4021.605434880" watchObservedRunningTime="2026-11-02 23:55:46.25291099 +0000 UTC m=+4021.605944366" Nov 02 23:55:46.952081 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:46.952035 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:46.954095 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:46.954054 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gd655 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 02 23:55:46.954262 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:46.954117 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" podUID="7701f578-a649-4f37-956b-aa1189848710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 02 23:55:47.952056 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:47.952015 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gd655 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 02 23:55:47.952283 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:47.952071 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" podUID="7701f578-a649-4f37-956b-aa1189848710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 02 23:55:48.951656 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:48.951619 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gd655 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 02 23:55:48.951858 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:48.951678 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" podUID="7701f578-a649-4f37-956b-aa1189848710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 02 23:55:49.952018 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:49.951977 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gd655 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 02 23:55:49.952564 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:49.952034 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" podUID="7701f578-a649-4f37-956b-aa1189848710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 02 23:55:50.952187 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:50.952146 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gd655 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 02 23:55:50.952666 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:50.952202 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" podUID="7701f578-a649-4f37-956b-aa1189848710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 02 23:55:51.951996 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:51.951957 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gd655 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 02 23:55:51.952231 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:51.952019 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" podUID="7701f578-a649-4f37-956b-aa1189848710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 02 23:55:52.846659 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:52.846622 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn"] Nov 02 23:55:52.850307 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:52.850285 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn" Nov 02 23:55:52.961516 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:52.955945 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gd655 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 02 23:55:52.961516 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:52.956025 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" podUID="7701f578-a649-4f37-956b-aa1189848710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 02 23:55:52.974243 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:52.974187 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3a1de1-dockercfg-99xdh\"" Nov 02 23:55:52.975281 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:52.974578 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3a1de1\"" Nov 02 23:55:52.992006 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:52.991962 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn"] Nov 02 23:55:53.013528 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:53.011336 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh"] Nov 02 23:55:53.018434 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:53.018405 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh" Nov 02 23:55:53.019239 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:53.019207 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e6016d-78a9-4a2c-b445-8e0a31415b6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn\" (UID: \"f7e6016d-78a9-4a2c-b445-8e0a31415b6a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn" Nov 02 23:55:53.019410 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:53.019391 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t972r\" (UniqueName: \"kubernetes.io/projected/f7e6016d-78a9-4a2c-b445-8e0a31415b6a-kube-api-access-t972r\") pod \"payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn\" (UID: \"f7e6016d-78a9-4a2c-b445-8e0a31415b6a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn" Nov 02 23:55:53.063652 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:53.063612 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh"] Nov 02 23:55:53.122353 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:53.122251 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-568cv\" (UniqueName: \"kubernetes.io/projected/aea387ce-efc0-404b-9f57-51a62c745736-kube-api-access-568cv\") pod \"payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh\" (UID: \"aea387ce-efc0-404b-9f57-51a62c745736\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh" Nov 02 23:55:53.122574 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:53.122379 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t972r\" (UniqueName: \"kubernetes.io/projected/f7e6016d-78a9-4a2c-b445-8e0a31415b6a-kube-api-access-t972r\") pod \"payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn\" (UID: \"f7e6016d-78a9-4a2c-b445-8e0a31415b6a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn" Nov 02 23:55:53.122574 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:53.122527 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea387ce-efc0-404b-9f57-51a62c745736-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh\" (UID: \"aea387ce-efc0-404b-9f57-51a62c745736\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh" Nov 02 23:55:53.122574 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:53.122563 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e6016d-78a9-4a2c-b445-8e0a31415b6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn\" (UID: \"f7e6016d-78a9-4a2c-b445-8e0a31415b6a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn" Nov 02 23:55:53.123339 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:53.123314 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e6016d-78a9-4a2c-b445-8e0a31415b6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn\" (UID: \"f7e6016d-78a9-4a2c-b445-8e0a31415b6a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn" Nov 02 23:55:53.183121 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:53.183085 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t972r\" (UniqueName: \"kubernetes.io/projected/f7e6016d-78a9-4a2c-b445-8e0a31415b6a-kube-api-access-t972r\") pod \"payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn\" (UID: \"f7e6016d-78a9-4a2c-b445-8e0a31415b6a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn" Nov 02 23:55:53.223576 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:53.223542 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea387ce-efc0-404b-9f57-51a62c745736-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh\" (UID: \"aea387ce-efc0-404b-9f57-51a62c745736\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh" Nov 02 23:55:53.223750 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:53.223660 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-568cv\" (UniqueName: \"kubernetes.io/projected/aea387ce-efc0-404b-9f57-51a62c745736-kube-api-access-568cv\") pod \"payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh\" (UID: \"aea387ce-efc0-404b-9f57-51a62c745736\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh" Nov 02 23:55:53.224312 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:53.224292 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea387ce-efc0-404b-9f57-51a62c745736-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh\" (UID: \"aea387ce-efc0-404b-9f57-51a62c745736\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh" Nov 02 23:55:53.284182 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:53.284147 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-568cv\" (UniqueName: \"kubernetes.io/projected/aea387ce-efc0-404b-9f57-51a62c745736-kube-api-access-568cv\") pod \"payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh\" (UID: \"aea387ce-efc0-404b-9f57-51a62c745736\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh" Nov 02 23:55:53.340343 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:53.340304 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh" Nov 02 23:55:53.465635 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:53.465603 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn" Nov 02 23:55:53.725426 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:53.725394 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh"] Nov 02 23:55:53.727870 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:55:53.727842 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea387ce_efc0_404b_9f57_51a62c745736.slice/crio-d92370b9ed295347707cec0e887db1c99e20edbf98a95f9c6f2808e6c7126b18 WatchSource:0}: Error finding container d92370b9ed295347707cec0e887db1c99e20edbf98a95f9c6f2808e6c7126b18: Status 404 returned error can't find the container with id d92370b9ed295347707cec0e887db1c99e20edbf98a95f9c6f2808e6c7126b18 Nov 02 23:55:53.846920 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:53.846883 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn"] Nov 02 23:55:53.847876 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:55:53.847847 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e6016d_78a9_4a2c_b445_8e0a31415b6a.slice/crio-d30ce0225772d6b7d505190a36b9ca338e86f341b89447774ff0bb2717f57918 WatchSource:0}: Error finding container d30ce0225772d6b7d505190a36b9ca338e86f341b89447774ff0bb2717f57918: Status 404 returned error can't find the container with id d30ce0225772d6b7d505190a36b9ca338e86f341b89447774ff0bb2717f57918 Nov 02 23:55:54.191058 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:54.190994 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn" event={"ID":"f7e6016d-78a9-4a2c-b445-8e0a31415b6a","Type":"ContainerStarted","Data":"aff3982cc265148f3e95dbd00337e274af33620ef11d3b731cfec8fa5b102d40"} Nov 02 23:55:54.191622 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:54.191094 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn" event={"ID":"f7e6016d-78a9-4a2c-b445-8e0a31415b6a","Type":"ContainerStarted","Data":"d30ce0225772d6b7d505190a36b9ca338e86f341b89447774ff0bb2717f57918"} Nov 02 23:55:54.191622 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:54.191162 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn" Nov 02 23:55:54.193699 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:54.193665 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh" event={"ID":"aea387ce-efc0-404b-9f57-51a62c745736","Type":"ContainerStarted","Data":"21b1e6c33ae3b4de9820f7cda7686853cbfe799299b3afa8cd9a2231a4feeda1"} Nov 02 23:55:54.193850 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:54.193731 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh" event={"ID":"aea387ce-efc0-404b-9f57-51a62c745736","Type":"ContainerStarted","Data":"d92370b9ed295347707cec0e887db1c99e20edbf98a95f9c6f2808e6c7126b18"} Nov 02 23:55:54.193850 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:54.193766 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh" Nov 02 23:55:54.306322 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:54.306241 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn" podStartSLOduration=2.306226678 podStartE2EDuration="2.306226678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:55:54.305378925 +0000 UTC m=+4029.658412300" watchObservedRunningTime="2026-11-02 23:55:54.306226678 +0000 UTC m=+4029.659260053" Nov 02 23:55:54.400192 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:54.400159 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:54.400566 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:54.400529 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:54.401672 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:54.401646 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 02 23:55:54.420000 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:54.419941 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh" podStartSLOduration=2.419923407 podStartE2EDuration="2.419923407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:55:54.419293942 +0000 UTC m=+4029.772327354" watchObservedRunningTime="2026-11-02 23:55:54.419923407 +0000 UTC m=+4029.772956784" Nov 02 23:55:58.046742 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:55:58.046703 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4"] Nov 02 23:56:05.203958 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:05.203925 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh" Nov 02 23:56:05.204445 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:05.204004 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn" Nov 02 23:56:07.585964 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:56:07.585924 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:56:11.914595 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:11.914550 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254"] Nov 02 23:56:11.934786 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:11.930420 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" Nov 02 23:56:11.959033 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:11.958986 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce392789-cafb-4d1a-ac0e-05578d20785d-metrics-certs\") pod \"maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254\" (UID: \"ce392789-cafb-4d1a-ac0e-05578d20785d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" Nov 02 23:56:11.959221 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:11.959198 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9r4l\" (UniqueName: \"kubernetes.io/projected/ce392789-cafb-4d1a-ac0e-05578d20785d-kube-api-access-b9r4l\") pod \"maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254\" (UID: \"ce392789-cafb-4d1a-ac0e-05578d20785d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" Nov 02 23:56:11.959289 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:11.959275 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce392789-cafb-4d1a-ac0e-05578d20785d-maas-api-tls\") pod \"maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254\" (UID: \"ce392789-cafb-4d1a-ac0e-05578d20785d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" Nov 02 23:56:11.959432 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:11.959412 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce392789-cafb-4d1a-ac0e-05578d20785d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254\" (UID: \"ce392789-cafb-4d1a-ac0e-05578d20785d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" Nov 02 23:56:11.971223 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:11.971185 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254"] Nov 02 23:56:12.017186 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:12.017143 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-5db41e-serving-cert\"" Nov 02 23:56:12.060936 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:12.060894 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9r4l\" (UniqueName: \"kubernetes.io/projected/ce392789-cafb-4d1a-ac0e-05578d20785d-kube-api-access-b9r4l\") pod \"maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254\" (UID: \"ce392789-cafb-4d1a-ac0e-05578d20785d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" Nov 02 23:56:12.061150 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:12.060960 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce392789-cafb-4d1a-ac0e-05578d20785d-maas-api-tls\") pod \"maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254\" (UID: \"ce392789-cafb-4d1a-ac0e-05578d20785d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" Nov 02 23:56:12.061150 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:12.061042 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce392789-cafb-4d1a-ac0e-05578d20785d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254\" (UID: \"ce392789-cafb-4d1a-ac0e-05578d20785d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" Nov 02 23:56:12.061150 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:56:12.061112 5114 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-serving-cert: secret "maas-api-e2e-inf-a-5db41e-serving-cert" not found Nov 02 23:56:12.061342 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:12.061150 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce392789-cafb-4d1a-ac0e-05578d20785d-metrics-certs\") pod \"maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254\" (UID: \"ce392789-cafb-4d1a-ac0e-05578d20785d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" Nov 02 23:56:12.061342 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:56:12.061205 5114 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce392789-cafb-4d1a-ac0e-05578d20785d-maas-api-tls podName:ce392789-cafb-4d1a-ac0e-05578d20785d nodeName:}" failed. No retries permitted until 2026-11-02 23:56:12.56118281 +0000 UTC m=+4047.914216186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ce392789-cafb-4d1a-ac0e-05578d20785d-maas-api-tls") pod "maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" (UID: "ce392789-cafb-4d1a-ac0e-05578d20785d") : secret "maas-api-e2e-inf-a-5db41e-serving-cert" not found Nov 02 23:56:12.061779 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:12.061760 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce392789-cafb-4d1a-ac0e-05578d20785d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254\" (UID: \"ce392789-cafb-4d1a-ac0e-05578d20785d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" Nov 02 23:56:12.064206 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:12.064180 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce392789-cafb-4d1a-ac0e-05578d20785d-metrics-certs\") pod \"maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254\" (UID: \"ce392789-cafb-4d1a-ac0e-05578d20785d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" Nov 02 23:56:12.124318 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:12.124289 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9r4l\" (UniqueName: \"kubernetes.io/projected/ce392789-cafb-4d1a-ac0e-05578d20785d-kube-api-access-b9r4l\") pod \"maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254\" (UID: \"ce392789-cafb-4d1a-ac0e-05578d20785d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" Nov 02 23:56:12.568060 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:12.568019 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce392789-cafb-4d1a-ac0e-05578d20785d-maas-api-tls\") pod \"maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254\" (UID: \"ce392789-cafb-4d1a-ac0e-05578d20785d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" Nov 02 23:56:12.570949 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:12.570915 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce392789-cafb-4d1a-ac0e-05578d20785d-maas-api-tls\") pod \"maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254\" (UID: \"ce392789-cafb-4d1a-ac0e-05578d20785d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" Nov 02 23:56:12.856121 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:12.856019 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" Nov 02 23:56:13.069988 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.069916 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" podUID="acd46b0c-b1e7-465e-a871-bef07e0f5e4e" containerName="main" containerID="cri-o://4d7926b1e13759743ba7e6da554ec0d68a0d71e03b2dc6c000cf233453f51a19" gracePeriod=45 Nov 02 23:56:13.166986 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.166896 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" podUID="acd46b0c-b1e7-465e-a871-bef07e0f5e4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.141:8000/ready\": dial tcp 10.134.0.141:8000: connect: connection refused" Nov 02 23:56:13.250508 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.250284 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254"] Nov 02 23:56:13.271805 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.271780 5114 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:56:13.323014 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.322761 5114 generic.go:356] "Generic (PLEG): container finished" podID="acd46b0c-b1e7-465e-a871-bef07e0f5e4e" containerID="4d7926b1e13759743ba7e6da554ec0d68a0d71e03b2dc6c000cf233453f51a19" exitCode=0 Nov 02 23:56:13.323014 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.322960 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" event={"ID":"acd46b0c-b1e7-465e-a871-bef07e0f5e4e","Type":"ContainerDied","Data":"4d7926b1e13759743ba7e6da554ec0d68a0d71e03b2dc6c000cf233453f51a19"} Nov 02 23:56:13.324933 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.324854 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" event={"ID":"ce392789-cafb-4d1a-ac0e-05578d20785d","Type":"ContainerStarted","Data":"a5a2749c7fc70870e0c3524f29727e0139910c36b096baeea104f0c7843e7535"} Nov 02 23:56:13.425974 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.425901 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:56:13.481309 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.481275 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-model-cache\") pod \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\" (UID: \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\") " Nov 02 23:56:13.481309 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.481310 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-dshm\") pod \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\" (UID: \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\") " Nov 02 23:56:13.481593 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.481390 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-kube-api-access-hgl27\" (UniqueName: \"kubernetes.io/projected/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-kube-api-access-hgl27\") pod \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\" (UID: \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\") " Nov 02 23:56:13.481593 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.481432 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-tmp-dir\") pod \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\" (UID: \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\") " Nov 02 23:56:13.481593 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.481504 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-tls-certs\" (UniqueName: \"kubernetes.io/secret/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-tls-certs\") pod \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\" (UID: \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\") " Nov 02 23:56:13.481771 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.481647 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-home\" (UniqueName: \"kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-home\") pod \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\" (UID: \"acd46b0c-b1e7-465e-a871-bef07e0f5e4e\") " Nov 02 23:56:13.481771 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.481690 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-model-cache" pod "acd46b0c-b1e7-465e-a871-bef07e0f5e4e" (UID: "acd46b0c-b1e7-465e-a871-bef07e0f5e4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:56:13.481771 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.481725 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-tmp-dir" pod "acd46b0c-b1e7-465e-a871-bef07e0f5e4e" (UID: "acd46b0c-b1e7-465e-a871-bef07e0f5e4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:56:13.481946 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.481920 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-home" pod "acd46b0c-b1e7-465e-a871-bef07e0f5e4e" (UID: "acd46b0c-b1e7-465e-a871-bef07e0f5e4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:56:13.482416 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.482394 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:56:13.482527 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.482423 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:56:13.482527 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.482438 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:56:13.484009 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.483986 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-dshm" pod "acd46b0c-b1e7-465e-a871-bef07e0f5e4e" (UID: "acd46b0c-b1e7-465e-a871-bef07e0f5e4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:56:13.484009 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.483992 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-kube-api-access-hgl27" pod "acd46b0c-b1e7-465e-a871-bef07e0f5e4e" (UID: "acd46b0c-b1e7-465e-a871-bef07e0f5e4e"). InnerVolumeSpecName "kube-api-access-hgl27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:56:13.484149 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.484050 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-tls-certs" pod "acd46b0c-b1e7-465e-a871-bef07e0f5e4e" (UID: "acd46b0c-b1e7-465e-a871-bef07e0f5e4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:56:13.583569 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.583536 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:56:13.583569 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.583568 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgl27\" (UniqueName: \"kubernetes.io/projected/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-kube-api-access-hgl27\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:56:13.583788 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:13.583592 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acd46b0c-b1e7-465e-a871-bef07e0f5e4e-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:56:14.332081 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:14.332050 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" Nov 02 23:56:14.332081 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:14.332066 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4" event={"ID":"acd46b0c-b1e7-465e-a871-bef07e0f5e4e","Type":"ContainerDied","Data":"14a37d04433c264ba16a38d2b76f3fb9cee173c942bf6002809f3cbe8f4e80d1"} Nov 02 23:56:14.332649 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:14.332118 5114 scope.go:122] "RemoveContainer" containerID="4d7926b1e13759743ba7e6da554ec0d68a0d71e03b2dc6c000cf233453f51a19" Nov 02 23:56:14.334008 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:14.333937 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" event={"ID":"ce392789-cafb-4d1a-ac0e-05578d20785d","Type":"ContainerStarted","Data":"8326a6f74ca049756bd7060007c1d56d34aa3f1c856b82dc646a33a51a634bb9"} Nov 02 23:56:14.334144 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:14.334134 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" Nov 02 23:56:14.504615 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:14.504554 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" podStartSLOduration=2.6900824869999997 podStartE2EDuration="3.504539874s" totalImagesPullingTime="814.457387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:56:14.50266249 +0000 UTC m=+4049.855695865" watchObservedRunningTime="2026-11-02 23:56:14.504539874 +0000 UTC m=+4049.857573248" Nov 02 23:56:14.613814 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:14.613710 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4"] Nov 02 23:56:14.617958 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:14.617920 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-6c3d8e/auth-test-model-6c3d8e-kserve-55f48f6b45-wlks4"] Nov 02 23:56:14.795099 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:14.795062 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acd46b0c-b1e7-465e-a871-bef07e0f5e4e" path="/var/lib/kubelet/pods/acd46b0c-b1e7-465e-a871-bef07e0f5e4e/volumes" Nov 02 23:56:20.348853 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:56:20.348812 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" Nov 02 23:56:37.586616 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:56:37.586526 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:57:07.585200 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:57:07.585161 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:57:12.701789 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:57:12.701743 5114 generic.go:356] "Generic (PLEG): container finished" podID="a53ad7b3-45b4-4450-a3c7-b7e00596d28d" containerID="c8f3b0709d5781d8731b8bbed90346192d2c62de79b750a46ff9b1a1840f953e" exitCode=0 Nov 02 23:57:12.702255 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:57:12.701813 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" event={"ID":"a53ad7b3-45b4-4450-a3c7-b7e00596d28d","Type":"ContainerDied","Data":"c8f3b0709d5781d8731b8bbed90346192d2c62de79b750a46ff9b1a1840f953e"} Nov 02 23:57:12.702255 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:57:12.702244 5114 scope.go:122] "RemoveContainer" containerID="c8f3b0709d5781d8731b8bbed90346192d2c62de79b750a46ff9b1a1840f953e" Nov 02 23:57:12.704082 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:57:12.704046 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:57:12.704203 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:57:12.704124 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:57:12.704203 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:57:12.704157 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:57:13.733447 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:57:13.733406 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" event={"ID":"a53ad7b3-45b4-4450-a3c7-b7e00596d28d","Type":"ContainerStarted","Data":"93e5af1f99cbe6dcbb31a4fa2206feb2229626b5af523ef14a25cc8836723ae2"} Nov 02 23:57:14.570124 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:57:14.570080 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:57:14.570330 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:57:14.570304 5114 patch_prober.go:29] interesting pod/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 02 23:57:14.570386 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:57:14.570356 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" podUID="a53ad7b3-45b4-4450-a3c7-b7e00596d28d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 02 23:57:15.569563 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:57:15.569521 5114 patch_prober.go:29] interesting pod/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 02 23:57:15.570049 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:57:15.569587 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" podUID="a53ad7b3-45b4-4450-a3c7-b7e00596d28d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 02 23:57:16.570320 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:57:16.570283 5114 patch_prober.go:29] interesting pod/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 02 23:57:16.570820 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:57:16.570340 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" podUID="a53ad7b3-45b4-4450-a3c7-b7e00596d28d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 02 23:57:17.569984 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:57:17.569943 5114 patch_prober.go:29] interesting pod/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 02 23:57:17.570188 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:57:17.570002 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" podUID="a53ad7b3-45b4-4450-a3c7-b7e00596d28d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 02 23:57:18.570514 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:57:18.570449 5114 patch_prober.go:29] interesting pod/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 02 23:57:18.570992 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:57:18.570529 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" podUID="a53ad7b3-45b4-4450-a3c7-b7e00596d28d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 02 23:57:19.574380 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:57:19.574346 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:57:19.574947 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:57:19.574775 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:57:19.575510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:57:19.575476 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 02 23:57:37.585700 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:57:37.585659 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:57:46.149799 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:57:46.149764 5114 scope.go:122] "RemoveContainer" containerID="a71a4a5156e524e48122c5870bed2c7b3ddd11f91be914e3dd7e37902f2008dd" Nov 02 23:57:58.959881 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:57:58.959836 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r"] Nov 02 23:57:59.070180 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:57:59.070142 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j"] Nov 02 23:58:03.960716 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:03.960664 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r" podUID="82af27ef-235f-4b77-bc32-69374be6e05e" containerName="payload-processing" containerID="cri-o://3fb2235f7dcd1a541579c906468f01c65b1ab16728c79b3c56e9f9b0af9c68d4" gracePeriod=25 Nov 02 23:58:04.070811 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:04.070752 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j" podUID="4aaa2b45-4263-485e-a86b-2148f4454cbe" containerName="payload-pre-processing" containerID="cri-o://a27394f5692b6fc3261de8ccca7279dff63d928cd3bd05fd878517e075834478" gracePeriod=25 Nov 02 23:58:04.230977 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:04.230908 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r" Nov 02 23:58:04.284354 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:04.284317 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82af27ef-235f-4b77-bc32-69374be6e05e-kube-api-access-f5qrm\" (UniqueName: \"kubernetes.io/projected/82af27ef-235f-4b77-bc32-69374be6e05e-kube-api-access-f5qrm\") pod \"82af27ef-235f-4b77-bc32-69374be6e05e\" (UID: \"82af27ef-235f-4b77-bc32-69374be6e05e\") " Nov 02 23:58:04.284627 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:04.284543 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82af27ef-235f-4b77-bc32-69374be6e05e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82af27ef-235f-4b77-bc32-69374be6e05e-plugins-config-volume\") pod \"82af27ef-235f-4b77-bc32-69374be6e05e\" (UID: \"82af27ef-235f-4b77-bc32-69374be6e05e\") " Nov 02 23:58:04.285725 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:04.285697 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82af27ef-235f-4b77-bc32-69374be6e05e-plugins-config-volume" pod "82af27ef-235f-4b77-bc32-69374be6e05e" (UID: "82af27ef-235f-4b77-bc32-69374be6e05e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:58:04.289170 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:04.289136 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82af27ef-235f-4b77-bc32-69374be6e05e-kube-api-access-f5qrm" pod "82af27ef-235f-4b77-bc32-69374be6e05e" (UID: "82af27ef-235f-4b77-bc32-69374be6e05e"). InnerVolumeSpecName "kube-api-access-f5qrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:58:04.329658 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:04.329632 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j" Nov 02 23:58:04.386250 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:04.386218 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aaa2b45-4263-485e-a86b-2148f4454cbe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aaa2b45-4263-485e-a86b-2148f4454cbe-plugins-config-volume\") pod \"4aaa2b45-4263-485e-a86b-2148f4454cbe\" (UID: \"4aaa2b45-4263-485e-a86b-2148f4454cbe\") " Nov 02 23:58:04.386441 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:04.386392 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aaa2b45-4263-485e-a86b-2148f4454cbe-kube-api-access-9trkz\" (UniqueName: \"kubernetes.io/projected/4aaa2b45-4263-485e-a86b-2148f4454cbe-kube-api-access-9trkz\") pod \"4aaa2b45-4263-485e-a86b-2148f4454cbe\" (UID: \"4aaa2b45-4263-485e-a86b-2148f4454cbe\") " Nov 02 23:58:04.386756 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:04.386729 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aaa2b45-4263-485e-a86b-2148f4454cbe-plugins-config-volume" pod "4aaa2b45-4263-485e-a86b-2148f4454cbe" (UID: "4aaa2b45-4263-485e-a86b-2148f4454cbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:58:04.386947 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:04.386924 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5qrm\" (UniqueName: \"kubernetes.io/projected/82af27ef-235f-4b77-bc32-69374be6e05e-kube-api-access-f5qrm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:04.387015 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:04.386954 5114 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aaa2b45-4263-485e-a86b-2148f4454cbe-plugins-config-volume\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:04.387015 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:04.386969 5114 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82af27ef-235f-4b77-bc32-69374be6e05e-plugins-config-volume\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:04.388785 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:04.388759 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aaa2b45-4263-485e-a86b-2148f4454cbe-kube-api-access-9trkz" pod "4aaa2b45-4263-485e-a86b-2148f4454cbe" (UID: "4aaa2b45-4263-485e-a86b-2148f4454cbe"). InnerVolumeSpecName "kube-api-access-9trkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:58:04.488301 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:04.488216 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9trkz\" (UniqueName: \"kubernetes.io/projected/4aaa2b45-4263-485e-a86b-2148f4454cbe-kube-api-access-9trkz\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:05.065376 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:05.065276 5114 generic.go:356] "Generic (PLEG): container finished" podID="4aaa2b45-4263-485e-a86b-2148f4454cbe" containerID="a27394f5692b6fc3261de8ccca7279dff63d928cd3bd05fd878517e075834478" exitCode=0 Nov 02 23:58:05.065887 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:05.065379 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j" Nov 02 23:58:05.065887 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:05.065412 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j" event={"ID":"4aaa2b45-4263-485e-a86b-2148f4454cbe","Type":"ContainerDied","Data":"a27394f5692b6fc3261de8ccca7279dff63d928cd3bd05fd878517e075834478"} Nov 02 23:58:05.065887 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:05.065499 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j" event={"ID":"4aaa2b45-4263-485e-a86b-2148f4454cbe","Type":"ContainerDied","Data":"43a875735658da63b0a2a9c456598d0d7af45c0e35d88afec09aacd5152c64a9"} Nov 02 23:58:05.065887 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:05.065527 5114 scope.go:122] "RemoveContainer" containerID="a27394f5692b6fc3261de8ccca7279dff63d928cd3bd05fd878517e075834478" Nov 02 23:58:05.067181 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:05.067156 5114 generic.go:356] "Generic (PLEG): container finished" podID="82af27ef-235f-4b77-bc32-69374be6e05e" containerID="3fb2235f7dcd1a541579c906468f01c65b1ab16728c79b3c56e9f9b0af9c68d4" exitCode=0 Nov 02 23:58:05.067284 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:05.067211 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r" Nov 02 23:58:05.067838 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:05.067456 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r" event={"ID":"82af27ef-235f-4b77-bc32-69374be6e05e","Type":"ContainerDied","Data":"3fb2235f7dcd1a541579c906468f01c65b1ab16728c79b3c56e9f9b0af9c68d4"} Nov 02 23:58:05.067838 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:05.067559 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r" event={"ID":"82af27ef-235f-4b77-bc32-69374be6e05e","Type":"ContainerDied","Data":"136deb78256e085dd0080187f2e46a638f065b2128b6ac00d5fc44f99fdac539"} Nov 02 23:58:05.076741 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:05.076719 5114 scope.go:122] "RemoveContainer" containerID="a27394f5692b6fc3261de8ccca7279dff63d928cd3bd05fd878517e075834478" Nov 02 23:58:05.077106 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:58:05.077086 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a27394f5692b6fc3261de8ccca7279dff63d928cd3bd05fd878517e075834478\": container with ID starting with a27394f5692b6fc3261de8ccca7279dff63d928cd3bd05fd878517e075834478 not found: ID does not exist" containerID="a27394f5692b6fc3261de8ccca7279dff63d928cd3bd05fd878517e075834478" Nov 02 23:58:05.077152 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:05.077118 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a27394f5692b6fc3261de8ccca7279dff63d928cd3bd05fd878517e075834478"} err="failed to get container status \"a27394f5692b6fc3261de8ccca7279dff63d928cd3bd05fd878517e075834478\": rpc error: code = NotFound desc = could not find container \"a27394f5692b6fc3261de8ccca7279dff63d928cd3bd05fd878517e075834478\": container with ID starting with a27394f5692b6fc3261de8ccca7279dff63d928cd3bd05fd878517e075834478 not found: ID does not exist" Nov 02 23:58:05.077152 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:05.077138 5114 scope.go:122] "RemoveContainer" containerID="3fb2235f7dcd1a541579c906468f01c65b1ab16728c79b3c56e9f9b0af9c68d4" Nov 02 23:58:05.088457 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:05.088430 5114 scope.go:122] "RemoveContainer" containerID="3fb2235f7dcd1a541579c906468f01c65b1ab16728c79b3c56e9f9b0af9c68d4" Nov 02 23:58:05.088829 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:58:05.088805 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fb2235f7dcd1a541579c906468f01c65b1ab16728c79b3c56e9f9b0af9c68d4\": container with ID starting with 3fb2235f7dcd1a541579c906468f01c65b1ab16728c79b3c56e9f9b0af9c68d4 not found: ID does not exist" containerID="3fb2235f7dcd1a541579c906468f01c65b1ab16728c79b3c56e9f9b0af9c68d4" Nov 02 23:58:05.088921 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:05.088838 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fb2235f7dcd1a541579c906468f01c65b1ab16728c79b3c56e9f9b0af9c68d4"} err="failed to get container status \"3fb2235f7dcd1a541579c906468f01c65b1ab16728c79b3c56e9f9b0af9c68d4\": rpc error: code = NotFound desc = could not find container \"3fb2235f7dcd1a541579c906468f01c65b1ab16728c79b3c56e9f9b0af9c68d4\": container with ID starting with 3fb2235f7dcd1a541579c906468f01c65b1ab16728c79b3c56e9f9b0af9c68d4 not found: ID does not exist" Nov 02 23:58:05.237087 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:05.237046 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r"] Nov 02 23:58:05.288054 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:05.288007 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5db41e-656b94f76c-5c72r"] Nov 02 23:58:05.348976 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:05.348871 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j"] Nov 02 23:58:05.352623 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:05.352585 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5db41e-74c699c779-5rd9j"] Nov 02 23:58:06.794512 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:06.794460 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aaa2b45-4263-485e-a86b-2148f4454cbe" path="/var/lib/kubelet/pods/4aaa2b45-4263-485e-a86b-2148f4454cbe/volumes" Nov 02 23:58:06.794938 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:06.794923 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82af27ef-235f-4b77-bc32-69374be6e05e" path="/var/lib/kubelet/pods/82af27ef-235f-4b77-bc32-69374be6e05e/volumes" Nov 02 23:58:07.584230 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:58:07.584189 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:58:37.584865 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:58:37.584817 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:58:42.057250 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:42.057203 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e130b395-v4lgm"] Nov 02 23:58:42.106579 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:42.106538 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e130b395-v4lgm"] Nov 02 23:58:42.793604 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:42.793567 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7c9c6f1-eb52-423a-9284-496cfbbb7cad" path="/var/lib/kubelet/pods/d7c9c6f1-eb52-423a-9284-496cfbbb7cad/volumes" Nov 02 23:58:52.357172 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:52.357125 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh"] Nov 02 23:58:52.464004 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:52.463963 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn"] Nov 02 23:58:57.357642 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:57.357589 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh" podUID="aea387ce-efc0-404b-9f57-51a62c745736" containerName="payload-processing" containerID="cri-o://21b1e6c33ae3b4de9820f7cda7686853cbfe799299b3afa8cd9a2231a4feeda1" gracePeriod=25 Nov 02 23:58:57.465496 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:57.465428 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn" podUID="f7e6016d-78a9-4a2c-b445-8e0a31415b6a" containerName="payload-pre-processing" containerID="cri-o://aff3982cc265148f3e95dbd00337e274af33620ef11d3b731cfec8fa5b102d40" gracePeriod=25 Nov 02 23:58:57.625940 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:57.625873 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh" Nov 02 23:58:57.731567 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:57.731538 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn" Nov 02 23:58:57.762055 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:57.762019 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aea387ce-efc0-404b-9f57-51a62c745736-kube-api-access-568cv\" (UniqueName: \"kubernetes.io/projected/aea387ce-efc0-404b-9f57-51a62c745736-kube-api-access-568cv\") pod \"aea387ce-efc0-404b-9f57-51a62c745736\" (UID: \"aea387ce-efc0-404b-9f57-51a62c745736\") " Nov 02 23:58:57.762272 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:57.762137 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aea387ce-efc0-404b-9f57-51a62c745736-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea387ce-efc0-404b-9f57-51a62c745736-plugins-config-volume\") pod \"aea387ce-efc0-404b-9f57-51a62c745736\" (UID: \"aea387ce-efc0-404b-9f57-51a62c745736\") " Nov 02 23:58:57.762623 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:57.762598 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aea387ce-efc0-404b-9f57-51a62c745736-plugins-config-volume" pod "aea387ce-efc0-404b-9f57-51a62c745736" (UID: "aea387ce-efc0-404b-9f57-51a62c745736"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:58:57.764587 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:57.764559 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea387ce-efc0-404b-9f57-51a62c745736-kube-api-access-568cv" pod "aea387ce-efc0-404b-9f57-51a62c745736" (UID: "aea387ce-efc0-404b-9f57-51a62c745736"). InnerVolumeSpecName "kube-api-access-568cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:58:57.868230 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:57.868192 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7e6016d-78a9-4a2c-b445-8e0a31415b6a-kube-api-access-t972r\" (UniqueName: \"kubernetes.io/projected/f7e6016d-78a9-4a2c-b445-8e0a31415b6a-kube-api-access-t972r\") pod \"f7e6016d-78a9-4a2c-b445-8e0a31415b6a\" (UID: \"f7e6016d-78a9-4a2c-b445-8e0a31415b6a\") " Nov 02 23:58:57.868527 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:57.868506 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7e6016d-78a9-4a2c-b445-8e0a31415b6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e6016d-78a9-4a2c-b445-8e0a31415b6a-plugins-config-volume\") pod \"f7e6016d-78a9-4a2c-b445-8e0a31415b6a\" (UID: \"f7e6016d-78a9-4a2c-b445-8e0a31415b6a\") " Nov 02 23:58:57.869639 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:57.869613 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-568cv\" (UniqueName: \"kubernetes.io/projected/aea387ce-efc0-404b-9f57-51a62c745736-kube-api-access-568cv\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:57.869781 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:57.869768 5114 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea387ce-efc0-404b-9f57-51a62c745736-plugins-config-volume\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:57.871248 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:57.871214 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7e6016d-78a9-4a2c-b445-8e0a31415b6a-plugins-config-volume" pod "f7e6016d-78a9-4a2c-b445-8e0a31415b6a" (UID: "f7e6016d-78a9-4a2c-b445-8e0a31415b6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:58:57.875799 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:57.875592 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e6016d-78a9-4a2c-b445-8e0a31415b6a-kube-api-access-t972r" pod "f7e6016d-78a9-4a2c-b445-8e0a31415b6a" (UID: "f7e6016d-78a9-4a2c-b445-8e0a31415b6a"). InnerVolumeSpecName "kube-api-access-t972r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:58:57.971374 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:57.971287 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t972r\" (UniqueName: \"kubernetes.io/projected/f7e6016d-78a9-4a2c-b445-8e0a31415b6a-kube-api-access-t972r\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:57.971374 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:57.971321 5114 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e6016d-78a9-4a2c-b445-8e0a31415b6a-plugins-config-volume\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:58.399601 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:58.399565 5114 generic.go:356] "Generic (PLEG): container finished" podID="aea387ce-efc0-404b-9f57-51a62c745736" containerID="21b1e6c33ae3b4de9820f7cda7686853cbfe799299b3afa8cd9a2231a4feeda1" exitCode=0 Nov 02 23:58:58.400120 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:58.399639 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh" Nov 02 23:58:58.400120 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:58.399647 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh" event={"ID":"aea387ce-efc0-404b-9f57-51a62c745736","Type":"ContainerDied","Data":"21b1e6c33ae3b4de9820f7cda7686853cbfe799299b3afa8cd9a2231a4feeda1"} Nov 02 23:58:58.400120 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:58.399736 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh" event={"ID":"aea387ce-efc0-404b-9f57-51a62c745736","Type":"ContainerDied","Data":"d92370b9ed295347707cec0e887db1c99e20edbf98a95f9c6f2808e6c7126b18"} Nov 02 23:58:58.400120 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:58.399759 5114 scope.go:122] "RemoveContainer" containerID="21b1e6c33ae3b4de9820f7cda7686853cbfe799299b3afa8cd9a2231a4feeda1" Nov 02 23:58:58.401320 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:58.401134 5114 generic.go:356] "Generic (PLEG): container finished" podID="f7e6016d-78a9-4a2c-b445-8e0a31415b6a" containerID="aff3982cc265148f3e95dbd00337e274af33620ef11d3b731cfec8fa5b102d40" exitCode=0 Nov 02 23:58:58.401320 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:58.401262 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn" Nov 02 23:58:58.401464 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:58.401263 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn" event={"ID":"f7e6016d-78a9-4a2c-b445-8e0a31415b6a","Type":"ContainerDied","Data":"aff3982cc265148f3e95dbd00337e274af33620ef11d3b731cfec8fa5b102d40"} Nov 02 23:58:58.401464 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:58.401379 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn" event={"ID":"f7e6016d-78a9-4a2c-b445-8e0a31415b6a","Type":"ContainerDied","Data":"d30ce0225772d6b7d505190a36b9ca338e86f341b89447774ff0bb2717f57918"} Nov 02 23:58:58.411057 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:58.411029 5114 scope.go:122] "RemoveContainer" containerID="21b1e6c33ae3b4de9820f7cda7686853cbfe799299b3afa8cd9a2231a4feeda1" Nov 02 23:58:58.411412 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:58:58.411388 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21b1e6c33ae3b4de9820f7cda7686853cbfe799299b3afa8cd9a2231a4feeda1\": container with ID starting with 21b1e6c33ae3b4de9820f7cda7686853cbfe799299b3afa8cd9a2231a4feeda1 not found: ID does not exist" containerID="21b1e6c33ae3b4de9820f7cda7686853cbfe799299b3afa8cd9a2231a4feeda1" Nov 02 23:58:58.411553 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:58.411422 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b1e6c33ae3b4de9820f7cda7686853cbfe799299b3afa8cd9a2231a4feeda1"} err="failed to get container status \"21b1e6c33ae3b4de9820f7cda7686853cbfe799299b3afa8cd9a2231a4feeda1\": rpc error: code = NotFound desc = could not find container \"21b1e6c33ae3b4de9820f7cda7686853cbfe799299b3afa8cd9a2231a4feeda1\": container with ID starting with 21b1e6c33ae3b4de9820f7cda7686853cbfe799299b3afa8cd9a2231a4feeda1 not found: ID does not exist" Nov 02 23:58:58.411553 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:58.411450 5114 scope.go:122] "RemoveContainer" containerID="aff3982cc265148f3e95dbd00337e274af33620ef11d3b731cfec8fa5b102d40" Nov 02 23:58:58.422128 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:58.422106 5114 scope.go:122] "RemoveContainer" containerID="aff3982cc265148f3e95dbd00337e274af33620ef11d3b731cfec8fa5b102d40" Nov 02 23:58:58.422443 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:58:58.422415 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aff3982cc265148f3e95dbd00337e274af33620ef11d3b731cfec8fa5b102d40\": container with ID starting with aff3982cc265148f3e95dbd00337e274af33620ef11d3b731cfec8fa5b102d40 not found: ID does not exist" containerID="aff3982cc265148f3e95dbd00337e274af33620ef11d3b731cfec8fa5b102d40" Nov 02 23:58:58.422534 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:58.422453 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff3982cc265148f3e95dbd00337e274af33620ef11d3b731cfec8fa5b102d40"} err="failed to get container status \"aff3982cc265148f3e95dbd00337e274af33620ef11d3b731cfec8fa5b102d40\": rpc error: code = NotFound desc = could not find container \"aff3982cc265148f3e95dbd00337e274af33620ef11d3b731cfec8fa5b102d40\": container with ID starting with aff3982cc265148f3e95dbd00337e274af33620ef11d3b731cfec8fa5b102d40 not found: ID does not exist" Nov 02 23:58:58.563084 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:58.563053 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh"] Nov 02 23:58:58.612442 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:58.612407 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a1de1-7b9dcb78c5-sddvh"] Nov 02 23:58:58.675794 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:58.675694 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn"] Nov 02 23:58:58.679891 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:58.679858 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3a1de1-f45d5f8c-hdgmn"] Nov 02 23:58:58.794323 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:58.794289 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aea387ce-efc0-404b-9f57-51a62c745736" path="/var/lib/kubelet/pods/aea387ce-efc0-404b-9f57-51a62c745736/volumes" Nov 02 23:58:58.794838 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:58:58.794822 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7e6016d-78a9-4a2c-b445-8e0a31415b6a" path="/var/lib/kubelet/pods/f7e6016d-78a9-4a2c-b445-8e0a31415b6a/volumes" Nov 02 23:59:04.362510 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:04.361647 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" podUID="fb16fc7d-3def-41dc-91a7-1103ccaf4c35" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:59:04.549052 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:04.547775 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" podUID="592a8e63-9584-4890-b38e-900b0482f735" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:59:07.583238 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:59:07.583199 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:59:11.961927 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:11.961891 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5feec3c6-z8vbl"] Nov 02 23:59:11.962604 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:11.962580 5114 cpu_manager.go:414] "containerMap: removing container" podUID="aea387ce-efc0-404b-9f57-51a62c745736" containerName="payload-processing" Nov 02 23:59:11.962720 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:11.962606 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aea387ce-efc0-404b-9f57-51a62c745736" containerName="payload-processing" Nov 02 23:59:11.962720 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:11.962636 5114 cpu_manager.go:414] "containerMap: removing container" podUID="82af27ef-235f-4b77-bc32-69374be6e05e" containerName="payload-processing" Nov 02 23:59:11.962720 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:11.962644 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82af27ef-235f-4b77-bc32-69374be6e05e" containerName="payload-processing" Nov 02 23:59:11.962720 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:11.962656 5114 cpu_manager.go:414] "containerMap: removing container" podUID="f7e6016d-78a9-4a2c-b445-8e0a31415b6a" containerName="payload-pre-processing" Nov 02 23:59:11.962720 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:11.962662 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7e6016d-78a9-4a2c-b445-8e0a31415b6a" containerName="payload-pre-processing" Nov 02 23:59:11.962720 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:11.962688 5114 cpu_manager.go:414] "containerMap: removing container" podUID="acd46b0c-b1e7-465e-a871-bef07e0f5e4e" containerName="main" Nov 02 23:59:11.962720 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:11.962695 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd46b0c-b1e7-465e-a871-bef07e0f5e4e" containerName="main" Nov 02 23:59:11.962720 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:11.962707 5114 cpu_manager.go:414] "containerMap: removing container" podUID="4aaa2b45-4263-485e-a86b-2148f4454cbe" containerName="payload-pre-processing" Nov 02 23:59:11.962720 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:11.962715 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aaa2b45-4263-485e-a86b-2148f4454cbe" containerName="payload-pre-processing" Nov 02 23:59:11.963172 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:11.962858 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd46b0c-b1e7-465e-a871-bef07e0f5e4e" containerName="main" Nov 02 23:59:11.963172 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:11.962872 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aaa2b45-4263-485e-a86b-2148f4454cbe" containerName="payload-pre-processing" Nov 02 23:59:11.963172 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:11.962887 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="aea387ce-efc0-404b-9f57-51a62c745736" containerName="payload-processing" Nov 02 23:59:11.963172 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:11.962898 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="82af27ef-235f-4b77-bc32-69374be6e05e" containerName="payload-processing" Nov 02 23:59:11.963172 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:11.962907 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7e6016d-78a9-4a2c-b445-8e0a31415b6a" containerName="payload-pre-processing" Nov 02 23:59:11.967649 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:11.967626 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feec3c6-z8vbl" Nov 02 23:59:12.023537 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:12.023503 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5feec3c6-z8vbl"] Nov 02 23:59:12.043531 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:12.036090 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7da2e976-6a1b-4ba5-8271-e10189770d6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5feec3c6-z8vbl\" (UID: \"7da2e976-6a1b-4ba5-8271-e10189770d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feec3c6-z8vbl" Nov 02 23:59:12.048368 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:12.044129 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g924v\" (UniqueName: \"kubernetes.io/projected/7da2e976-6a1b-4ba5-8271-e10189770d6a-kube-api-access-g924v\") pod \"maas-api-revoke-sub-5feec3c6-z8vbl\" (UID: \"7da2e976-6a1b-4ba5-8271-e10189770d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feec3c6-z8vbl" Nov 02 23:59:12.048636 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:12.048592 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7da2e976-6a1b-4ba5-8271-e10189770d6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5feec3c6-z8vbl\" (UID: \"7da2e976-6a1b-4ba5-8271-e10189770d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feec3c6-z8vbl" Nov 02 23:59:12.069473 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:12.069439 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:59:12.069681 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:12.069541 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:59:12.149750 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:12.149704 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7da2e976-6a1b-4ba5-8271-e10189770d6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5feec3c6-z8vbl\" (UID: \"7da2e976-6a1b-4ba5-8271-e10189770d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feec3c6-z8vbl" Nov 02 23:59:12.149975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:12.149789 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7da2e976-6a1b-4ba5-8271-e10189770d6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5feec3c6-z8vbl\" (UID: \"7da2e976-6a1b-4ba5-8271-e10189770d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feec3c6-z8vbl" Nov 02 23:59:12.149975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:12.149917 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g924v\" (UniqueName: \"kubernetes.io/projected/7da2e976-6a1b-4ba5-8271-e10189770d6a-kube-api-access-g924v\") pod \"maas-api-revoke-sub-5feec3c6-z8vbl\" (UID: \"7da2e976-6a1b-4ba5-8271-e10189770d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feec3c6-z8vbl" Nov 02 23:59:12.150577 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:12.150549 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7da2e976-6a1b-4ba5-8271-e10189770d6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5feec3c6-z8vbl\" (UID: \"7da2e976-6a1b-4ba5-8271-e10189770d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feec3c6-z8vbl" Nov 02 23:59:12.152440 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:12.152418 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7da2e976-6a1b-4ba5-8271-e10189770d6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5feec3c6-z8vbl\" (UID: \"7da2e976-6a1b-4ba5-8271-e10189770d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feec3c6-z8vbl" Nov 02 23:59:12.211252 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:12.211222 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g924v\" (UniqueName: \"kubernetes.io/projected/7da2e976-6a1b-4ba5-8271-e10189770d6a-kube-api-access-g924v\") pod \"maas-api-revoke-sub-5feec3c6-z8vbl\" (UID: \"7da2e976-6a1b-4ba5-8271-e10189770d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feec3c6-z8vbl" Nov 02 23:59:12.281260 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:12.281178 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feec3c6-z8vbl" Nov 02 23:59:12.660593 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:12.660566 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5feec3c6-z8vbl"] Nov 02 23:59:12.662340 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:59:12.662305 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da2e976_6a1b_4ba5_8271_e10189770d6a.slice/crio-431927a10594055d8d5a88cdd8b8cb11980ff2e2cc0fe9c1d69a53bab7de3c7d WatchSource:0}: Error finding container 431927a10594055d8d5a88cdd8b8cb11980ff2e2cc0fe9c1d69a53bab7de3c7d: Status 404 returned error can't find the container with id 431927a10594055d8d5a88cdd8b8cb11980ff2e2cc0fe9c1d69a53bab7de3c7d Nov 02 23:59:13.627660 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:13.627622 5114 generic.go:356] "Generic (PLEG): container finished" podID="7da2e976-6a1b-4ba5-8271-e10189770d6a" containerID="15eddbaa87f64d556ba0996b1af2693931305175f84627bc1157c04d0f3027c6" exitCode=0 Nov 02 23:59:13.628080 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:13.627706 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feec3c6-z8vbl" event={"ID":"7da2e976-6a1b-4ba5-8271-e10189770d6a","Type":"ContainerDied","Data":"15eddbaa87f64d556ba0996b1af2693931305175f84627bc1157c04d0f3027c6"} Nov 02 23:59:13.628080 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:13.627760 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feec3c6-z8vbl" event={"ID":"7da2e976-6a1b-4ba5-8271-e10189770d6a","Type":"ContainerStarted","Data":"431927a10594055d8d5a88cdd8b8cb11980ff2e2cc0fe9c1d69a53bab7de3c7d"} Nov 02 23:59:14.772046 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:14.772018 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feec3c6-z8vbl" Nov 02 23:59:14.882292 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:14.882256 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7da2e976-6a1b-4ba5-8271-e10189770d6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7da2e976-6a1b-4ba5-8271-e10189770d6a-maas-api-xks-ca\") pod \"7da2e976-6a1b-4ba5-8271-e10189770d6a\" (UID: \"7da2e976-6a1b-4ba5-8271-e10189770d6a\") " Nov 02 23:59:14.882532 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:14.882383 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7da2e976-6a1b-4ba5-8271-e10189770d6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7da2e976-6a1b-4ba5-8271-e10189770d6a-maas-api-openshift-ca\") pod \"7da2e976-6a1b-4ba5-8271-e10189770d6a\" (UID: \"7da2e976-6a1b-4ba5-8271-e10189770d6a\") " Nov 02 23:59:14.882616 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:14.882550 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7da2e976-6a1b-4ba5-8271-e10189770d6a-kube-api-access-g924v\" (UniqueName: \"kubernetes.io/projected/7da2e976-6a1b-4ba5-8271-e10189770d6a-kube-api-access-g924v\") pod \"7da2e976-6a1b-4ba5-8271-e10189770d6a\" (UID: \"7da2e976-6a1b-4ba5-8271-e10189770d6a\") " Nov 02 23:59:14.882801 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:14.882777 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7da2e976-6a1b-4ba5-8271-e10189770d6a-maas-api-openshift-ca" pod "7da2e976-6a1b-4ba5-8271-e10189770d6a" (UID: "7da2e976-6a1b-4ba5-8271-e10189770d6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:59:14.883031 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:14.883006 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7da2e976-6a1b-4ba5-8271-e10189770d6a-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:14.884879 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:14.884821 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7da2e976-6a1b-4ba5-8271-e10189770d6a-kube-api-access-g924v" pod "7da2e976-6a1b-4ba5-8271-e10189770d6a" (UID: "7da2e976-6a1b-4ba5-8271-e10189770d6a"). InnerVolumeSpecName "kube-api-access-g924v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:59:14.884973 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:14.884873 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7da2e976-6a1b-4ba5-8271-e10189770d6a-maas-api-xks-ca" pod "7da2e976-6a1b-4ba5-8271-e10189770d6a" (UID: "7da2e976-6a1b-4ba5-8271-e10189770d6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:59:14.983816 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:14.983780 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7da2e976-6a1b-4ba5-8271-e10189770d6a-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:14.983816 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:14.983812 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g924v\" (UniqueName: \"kubernetes.io/projected/7da2e976-6a1b-4ba5-8271-e10189770d6a-kube-api-access-g924v\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:15.644975 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:15.644934 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feec3c6-z8vbl" event={"ID":"7da2e976-6a1b-4ba5-8271-e10189770d6a","Type":"ContainerDied","Data":"431927a10594055d8d5a88cdd8b8cb11980ff2e2cc0fe9c1d69a53bab7de3c7d"} Nov 02 23:59:15.645195 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:15.644996 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="431927a10594055d8d5a88cdd8b8cb11980ff2e2cc0fe9c1d69a53bab7de3c7d" Nov 02 23:59:15.645195 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:15.644949 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feec3c6-z8vbl" Nov 02 23:59:22.907350 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:22.907309 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57"] Nov 02 23:59:22.907883 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:22.907794 5114 cpu_manager.go:414] "containerMap: removing container" podUID="7da2e976-6a1b-4ba5-8271-e10189770d6a" containerName="revoke-keys" Nov 02 23:59:22.907883 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:22.907807 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7da2e976-6a1b-4ba5-8271-e10189770d6a" containerName="revoke-keys" Nov 02 23:59:22.907973 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:22.907919 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="7da2e976-6a1b-4ba5-8271-e10189770d6a" containerName="revoke-keys" Nov 02 23:59:22.911285 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:22.911266 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57" Nov 02 23:59:22.968725 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:22.968682 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57"] Nov 02 23:59:22.990240 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:22.990198 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blhz5\" (UniqueName: \"kubernetes.io/projected/84a4d12d-1da0-494b-a957-85600b3842f4-kube-api-access-blhz5\") pod \"maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57\" (UID: \"84a4d12d-1da0-494b-a957-85600b3842f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57" Nov 02 23:59:22.990468 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:22.990319 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a4d12d-1da0-494b-a957-85600b3842f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57\" (UID: \"84a4d12d-1da0-494b-a957-85600b3842f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57" Nov 02 23:59:22.990468 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:22.990384 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a4d12d-1da0-494b-a957-85600b3842f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57\" (UID: \"84a4d12d-1da0-494b-a957-85600b3842f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57" Nov 02 23:59:23.014908 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:23.014870 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:59:23.015094 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:23.014955 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:59:23.091826 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:23.091786 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blhz5\" (UniqueName: \"kubernetes.io/projected/84a4d12d-1da0-494b-a957-85600b3842f4-kube-api-access-blhz5\") pod \"maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57\" (UID: \"84a4d12d-1da0-494b-a957-85600b3842f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57" Nov 02 23:59:23.092024 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:23.091882 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a4d12d-1da0-494b-a957-85600b3842f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57\" (UID: \"84a4d12d-1da0-494b-a957-85600b3842f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57" Nov 02 23:59:23.092024 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:23.091969 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a4d12d-1da0-494b-a957-85600b3842f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57\" (UID: \"84a4d12d-1da0-494b-a957-85600b3842f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57" Nov 02 23:59:23.092609 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:23.092583 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a4d12d-1da0-494b-a957-85600b3842f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57\" (UID: \"84a4d12d-1da0-494b-a957-85600b3842f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57" Nov 02 23:59:23.094568 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:23.094545 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a4d12d-1da0-494b-a957-85600b3842f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57\" (UID: \"84a4d12d-1da0-494b-a957-85600b3842f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57" Nov 02 23:59:23.153056 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:23.153020 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blhz5\" (UniqueName: \"kubernetes.io/projected/84a4d12d-1da0-494b-a957-85600b3842f4-kube-api-access-blhz5\") pod \"maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57\" (UID: \"84a4d12d-1da0-494b-a957-85600b3842f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57" Nov 02 23:59:23.225034 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:23.224935 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57" Nov 02 23:59:23.600732 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:23.600704 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57"] Nov 02 23:59:23.602671 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:59:23.602640 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a4d12d_1da0_494b_a957_85600b3842f4.slice/crio-c14f7a0436cf7f4ee0e95fecf514774050dd5b2762cc13773b6b5e7f44b64d5d WatchSource:0}: Error finding container c14f7a0436cf7f4ee0e95fecf514774050dd5b2762cc13773b6b5e7f44b64d5d: Status 404 returned error can't find the container with id c14f7a0436cf7f4ee0e95fecf514774050dd5b2762cc13773b6b5e7f44b64d5d Nov 02 23:59:23.693254 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:23.693214 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57" event={"ID":"84a4d12d-1da0-494b-a957-85600b3842f4","Type":"ContainerStarted","Data":"c14f7a0436cf7f4ee0e95fecf514774050dd5b2762cc13773b6b5e7f44b64d5d"} Nov 02 23:59:24.700313 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:24.700274 5114 generic.go:356] "Generic (PLEG): container finished" podID="84a4d12d-1da0-494b-a957-85600b3842f4" containerID="ae9b4e12c15b42b08b8f461f5f8f02a436d2213a5ffb46eee91822762b8f1b1a" exitCode=0 Nov 02 23:59:24.700830 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:24.700361 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57" event={"ID":"84a4d12d-1da0-494b-a957-85600b3842f4","Type":"ContainerDied","Data":"ae9b4e12c15b42b08b8f461f5f8f02a436d2213a5ffb46eee91822762b8f1b1a"} Nov 02 23:59:25.862737 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:25.862706 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57" Nov 02 23:59:25.921284 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:25.921243 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a4d12d-1da0-494b-a957-85600b3842f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a4d12d-1da0-494b-a957-85600b3842f4-maas-api-openshift-ca\") pod \"84a4d12d-1da0-494b-a957-85600b3842f4\" (UID: \"84a4d12d-1da0-494b-a957-85600b3842f4\") " Nov 02 23:59:25.921477 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:25.921348 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84a4d12d-1da0-494b-a957-85600b3842f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a4d12d-1da0-494b-a957-85600b3842f4-maas-api-xks-ca\") pod \"84a4d12d-1da0-494b-a957-85600b3842f4\" (UID: \"84a4d12d-1da0-494b-a957-85600b3842f4\") " Nov 02 23:59:25.921477 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:25.921404 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a4d12d-1da0-494b-a957-85600b3842f4-kube-api-access-blhz5\" (UniqueName: \"kubernetes.io/projected/84a4d12d-1da0-494b-a957-85600b3842f4-kube-api-access-blhz5\") pod \"84a4d12d-1da0-494b-a957-85600b3842f4\" (UID: \"84a4d12d-1da0-494b-a957-85600b3842f4\") " Nov 02 23:59:25.921751 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:25.921725 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a4d12d-1da0-494b-a957-85600b3842f4-maas-api-openshift-ca" pod "84a4d12d-1da0-494b-a957-85600b3842f4" (UID: "84a4d12d-1da0-494b-a957-85600b3842f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:59:25.921884 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:25.921869 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a4d12d-1da0-494b-a957-85600b3842f4-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:25.923790 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:25.923762 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a4d12d-1da0-494b-a957-85600b3842f4-maas-api-xks-ca" pod "84a4d12d-1da0-494b-a957-85600b3842f4" (UID: "84a4d12d-1da0-494b-a957-85600b3842f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:59:25.923890 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:25.923806 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a4d12d-1da0-494b-a957-85600b3842f4-kube-api-access-blhz5" pod "84a4d12d-1da0-494b-a957-85600b3842f4" (UID: "84a4d12d-1da0-494b-a957-85600b3842f4"). InnerVolumeSpecName "kube-api-access-blhz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:59:26.023576 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:26.023460 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a4d12d-1da0-494b-a957-85600b3842f4-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:26.023576 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:26.023526 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blhz5\" (UniqueName: \"kubernetes.io/projected/84a4d12d-1da0-494b-a957-85600b3842f4-kube-api-access-blhz5\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:26.713396 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:26.713355 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57" Nov 02 23:59:26.713618 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:26.713388 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57" event={"ID":"84a4d12d-1da0-494b-a957-85600b3842f4","Type":"ContainerDied","Data":"c14f7a0436cf7f4ee0e95fecf514774050dd5b2762cc13773b6b5e7f44b64d5d"} Nov 02 23:59:26.713618 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:26.713469 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c14f7a0436cf7f4ee0e95fecf514774050dd5b2762cc13773b6b5e7f44b64d5d" Nov 02 23:59:37.584847 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:59:37.584798 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 02 23:59:37.899802 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:37.899716 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf"] Nov 02 23:59:37.899994 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:37.899970 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" podUID="9ab21771-e42c-462f-b83f-ee47948059dd" containerName="istio-proxy" containerID="cri-o://7bdac0b4eeae75b594b4e076c78810016f127ef6de8b16b3b0a4e886896e4117" gracePeriod=30 Nov 02 23:59:43.165381 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.165313 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:59:43.229424 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.229383 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-istio-envoy\") pod \"9ab21771-e42c-462f-b83f-ee47948059dd\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " Nov 02 23:59:43.229424 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.229427 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-workload-certs\") pod \"9ab21771-e42c-462f-b83f-ee47948059dd\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " Nov 02 23:59:43.229729 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.229570 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-workload-socket\") pod \"9ab21771-e42c-462f-b83f-ee47948059dd\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " Nov 02 23:59:43.229729 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.229661 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ab21771-e42c-462f-b83f-ee47948059dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ab21771-e42c-462f-b83f-ee47948059dd-istio-podinfo\") pod \"9ab21771-e42c-462f-b83f-ee47948059dd\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " Nov 02 23:59:43.229729 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.229699 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ab21771-e42c-462f-b83f-ee47948059dd-istio-token\" (UniqueName: \"kubernetes.io/projected/9ab21771-e42c-462f-b83f-ee47948059dd-istio-token\") pod \"9ab21771-e42c-462f-b83f-ee47948059dd\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " Nov 02 23:59:43.229900 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.229753 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-credential-socket\") pod \"9ab21771-e42c-462f-b83f-ee47948059dd\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " Nov 02 23:59:43.229900 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.229765 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-workload-certs" pod "9ab21771-e42c-462f-b83f-ee47948059dd" (UID: "9ab21771-e42c-462f-b83f-ee47948059dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:59:43.229900 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.229791 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ab21771-e42c-462f-b83f-ee47948059dd-kube-api-access-lrnjf\" (UniqueName: \"kubernetes.io/projected/9ab21771-e42c-462f-b83f-ee47948059dd-kube-api-access-lrnjf\") pod \"9ab21771-e42c-462f-b83f-ee47948059dd\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " Nov 02 23:59:43.229900 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.229842 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-istio-data\") pod \"9ab21771-e42c-462f-b83f-ee47948059dd\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " Nov 02 23:59:43.229900 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.229891 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-workload-socket" pod "9ab21771-e42c-462f-b83f-ee47948059dd" (UID: "9ab21771-e42c-462f-b83f-ee47948059dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:59:43.230173 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.229911 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ab21771-e42c-462f-b83f-ee47948059dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ab21771-e42c-462f-b83f-ee47948059dd-istiod-ca-cert\") pod \"9ab21771-e42c-462f-b83f-ee47948059dd\" (UID: \"9ab21771-e42c-462f-b83f-ee47948059dd\") " Nov 02 23:59:43.230173 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.230086 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-credential-socket" pod "9ab21771-e42c-462f-b83f-ee47948059dd" (UID: "9ab21771-e42c-462f-b83f-ee47948059dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:59:43.230337 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.230153 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-istio-data" pod "9ab21771-e42c-462f-b83f-ee47948059dd" (UID: "9ab21771-e42c-462f-b83f-ee47948059dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:59:43.230337 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.230324 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ab21771-e42c-462f-b83f-ee47948059dd-istiod-ca-cert" pod "9ab21771-e42c-462f-b83f-ee47948059dd" (UID: "9ab21771-e42c-462f-b83f-ee47948059dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:59:43.230618 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.230583 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:43.230618 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.230619 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:43.230852 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.230633 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ab21771-e42c-462f-b83f-ee47948059dd-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:43.230852 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.230647 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:43.230852 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.230660 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:43.232284 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.232249 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-istio-envoy" pod "9ab21771-e42c-462f-b83f-ee47948059dd" (UID: "9ab21771-e42c-462f-b83f-ee47948059dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:59:43.232284 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.232248 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab21771-e42c-462f-b83f-ee47948059dd-istio-token" pod "9ab21771-e42c-462f-b83f-ee47948059dd" (UID: "9ab21771-e42c-462f-b83f-ee47948059dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:59:43.232420 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.232335 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab21771-e42c-462f-b83f-ee47948059dd-kube-api-access-lrnjf" pod "9ab21771-e42c-462f-b83f-ee47948059dd" (UID: "9ab21771-e42c-462f-b83f-ee47948059dd"). InnerVolumeSpecName "kube-api-access-lrnjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:59:43.232420 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.232369 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ab21771-e42c-462f-b83f-ee47948059dd-istio-podinfo" pod "9ab21771-e42c-462f-b83f-ee47948059dd" (UID: "9ab21771-e42c-462f-b83f-ee47948059dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:59:43.332184 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.332136 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrnjf\" (UniqueName: \"kubernetes.io/projected/9ab21771-e42c-462f-b83f-ee47948059dd-kube-api-access-lrnjf\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:43.332184 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.332177 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ab21771-e42c-462f-b83f-ee47948059dd-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:43.332184 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.332190 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ab21771-e42c-462f-b83f-ee47948059dd-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:43.332440 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.332203 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ab21771-e42c-462f-b83f-ee47948059dd-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:43.820348 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.820294 5114 generic.go:356] "Generic (PLEG): container finished" podID="9ab21771-e42c-462f-b83f-ee47948059dd" containerID="7bdac0b4eeae75b594b4e076c78810016f127ef6de8b16b3b0a4e886896e4117" exitCode=0 Nov 02 23:59:43.820615 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.820430 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" event={"ID":"9ab21771-e42c-462f-b83f-ee47948059dd","Type":"ContainerDied","Data":"7bdac0b4eeae75b594b4e076c78810016f127ef6de8b16b3b0a4e886896e4117"} Nov 02 23:59:43.820615 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.820514 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" event={"ID":"9ab21771-e42c-462f-b83f-ee47948059dd","Type":"ContainerDied","Data":"25c7af79c95ff3a227a8532e18e01600dfc5201671720f4ca3a6aeaedf8e9331"} Nov 02 23:59:43.820615 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.820550 5114 scope.go:122] "RemoveContainer" containerID="7bdac0b4eeae75b594b4e076c78810016f127ef6de8b16b3b0a4e886896e4117" Nov 02 23:59:43.820831 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.820744 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf" Nov 02 23:59:43.844761 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.844733 5114 scope.go:122] "RemoveContainer" containerID="defe1bd2fff79a70b520b761a6f75def4df0e187ed8b7b9b339e8c1d1e1bba1e" Nov 02 23:59:43.860589 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.860393 5114 scope.go:122] "RemoveContainer" containerID="7bdac0b4eeae75b594b4e076c78810016f127ef6de8b16b3b0a4e886896e4117" Nov 02 23:59:43.861341 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:59:43.860962 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bdac0b4eeae75b594b4e076c78810016f127ef6de8b16b3b0a4e886896e4117\": container with ID starting with 7bdac0b4eeae75b594b4e076c78810016f127ef6de8b16b3b0a4e886896e4117 not found: ID does not exist" containerID="7bdac0b4eeae75b594b4e076c78810016f127ef6de8b16b3b0a4e886896e4117" Nov 02 23:59:43.861341 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.861041 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bdac0b4eeae75b594b4e076c78810016f127ef6de8b16b3b0a4e886896e4117"} err="failed to get container status \"7bdac0b4eeae75b594b4e076c78810016f127ef6de8b16b3b0a4e886896e4117\": rpc error: code = NotFound desc = could not find container \"7bdac0b4eeae75b594b4e076c78810016f127ef6de8b16b3b0a4e886896e4117\": container with ID starting with 7bdac0b4eeae75b594b4e076c78810016f127ef6de8b16b3b0a4e886896e4117 not found: ID does not exist" Nov 02 23:59:43.861341 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.861073 5114 scope.go:122] "RemoveContainer" containerID="defe1bd2fff79a70b520b761a6f75def4df0e187ed8b7b9b339e8c1d1e1bba1e" Nov 02 23:59:43.861667 ip-10-0-135-250 kubenswrapper[5114]: E1102 23:59:43.861626 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"defe1bd2fff79a70b520b761a6f75def4df0e187ed8b7b9b339e8c1d1e1bba1e\": container with ID starting with defe1bd2fff79a70b520b761a6f75def4df0e187ed8b7b9b339e8c1d1e1bba1e not found: ID does not exist" containerID="defe1bd2fff79a70b520b761a6f75def4df0e187ed8b7b9b339e8c1d1e1bba1e" Nov 02 23:59:43.861744 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.861667 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"defe1bd2fff79a70b520b761a6f75def4df0e187ed8b7b9b339e8c1d1e1bba1e"} err="failed to get container status \"defe1bd2fff79a70b520b761a6f75def4df0e187ed8b7b9b339e8c1d1e1bba1e\": rpc error: code = NotFound desc = could not find container \"defe1bd2fff79a70b520b761a6f75def4df0e187ed8b7b9b339e8c1d1e1bba1e\": container with ID starting with defe1bd2fff79a70b520b761a6f75def4df0e187ed8b7b9b339e8c1d1e1bba1e not found: ID does not exist" Nov 02 23:59:43.989100 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:43.989054 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf"] Nov 02 23:59:44.039475 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:44.039422 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f7002-openshift-default-76d8d8f7d5-fqwsf"] Nov 02 23:59:44.825511 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:44.811386 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ab21771-e42c-462f-b83f-ee47948059dd" path="/var/lib/kubelet/pods/9ab21771-e42c-462f-b83f-ee47948059dd/volumes" Nov 02 23:59:46.196509 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:46.195848 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" podUID="5e3275e5-f49f-4ca7-8d49-03178a77fee6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 23:59:46.299605 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:46.268275 5114 scope.go:122] "RemoveContainer" containerID="ebbab4d6411fb6f63e6f73121899979b93cfcb4a74df2384eb7670bf528a4975" Nov 02 23:59:46.345698 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:46.345637 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" podUID="e9da41e7-2e4b-4912-9267-3f732f714155" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 23:59:48.045800 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:48.045759 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9946cb57-82svt"] Nov 02 23:59:48.095432 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:48.095394 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9946cb57-82svt"] Nov 02 23:59:48.792948 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:48.792904 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76db2b58-20d0-4538-9ef4-347e8a7f4301" path="/var/lib/kubelet/pods/76db2b58-20d0-4538-9ef4-347e8a7f4301/volumes" Nov 02 23:59:49.061886 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:49.061801 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f5da0f1-8q8fm"] Nov 02 23:59:49.062511 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:49.062455 5114 cpu_manager.go:414] "containerMap: removing container" podUID="9ab21771-e42c-462f-b83f-ee47948059dd" containerName="istio-proxy" Nov 02 23:59:49.062623 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:49.062513 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ab21771-e42c-462f-b83f-ee47948059dd" containerName="istio-proxy" Nov 02 23:59:49.062623 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:49.062548 5114 cpu_manager.go:414] "containerMap: removing container" podUID="84a4d12d-1da0-494b-a957-85600b3842f4" containerName="revoke-keys" Nov 02 23:59:49.062623 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:49.062558 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a4d12d-1da0-494b-a957-85600b3842f4" containerName="revoke-keys" Nov 02 23:59:49.062623 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:49.062570 5114 cpu_manager.go:414] "containerMap: removing container" podUID="9ab21771-e42c-462f-b83f-ee47948059dd" containerName="istio-proxy" Nov 02 23:59:49.062623 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:49.062578 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ab21771-e42c-462f-b83f-ee47948059dd" containerName="istio-proxy" Nov 02 23:59:49.062881 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:49.062736 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a4d12d-1da0-494b-a957-85600b3842f4" containerName="revoke-keys" Nov 02 23:59:49.062881 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:49.062749 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ab21771-e42c-462f-b83f-ee47948059dd" containerName="istio-proxy" Nov 02 23:59:49.067673 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:49.067648 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5da0f1-8q8fm" Nov 02 23:59:49.088665 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:49.088625 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1afacc90-2b81-42c2-86af-1622afdfca74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f5da0f1-8q8fm\" (UID: \"1afacc90-2b81-42c2-86af-1622afdfca74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5da0f1-8q8fm" Nov 02 23:59:49.088665 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:49.088665 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1afacc90-2b81-42c2-86af-1622afdfca74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f5da0f1-8q8fm\" (UID: \"1afacc90-2b81-42c2-86af-1622afdfca74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5da0f1-8q8fm" Nov 02 23:59:49.088925 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:49.088845 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hln88\" (UniqueName: \"kubernetes.io/projected/1afacc90-2b81-42c2-86af-1622afdfca74-kube-api-access-hln88\") pod \"maas-api-revoke-sub-7f5da0f1-8q8fm\" (UID: \"1afacc90-2b81-42c2-86af-1622afdfca74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5da0f1-8q8fm" Nov 02 23:59:49.119684 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:49.119646 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:59:49.169240 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:49.169202 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:59:49.171978 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:49.171938 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f5da0f1-8q8fm"] Nov 02 23:59:49.190166 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:49.190127 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1afacc90-2b81-42c2-86af-1622afdfca74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f5da0f1-8q8fm\" (UID: \"1afacc90-2b81-42c2-86af-1622afdfca74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5da0f1-8q8fm" Nov 02 23:59:49.190166 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:49.190175 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1afacc90-2b81-42c2-86af-1622afdfca74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f5da0f1-8q8fm\" (UID: \"1afacc90-2b81-42c2-86af-1622afdfca74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5da0f1-8q8fm" Nov 02 23:59:49.190437 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:49.190378 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hln88\" (UniqueName: \"kubernetes.io/projected/1afacc90-2b81-42c2-86af-1622afdfca74-kube-api-access-hln88\") pod \"maas-api-revoke-sub-7f5da0f1-8q8fm\" (UID: \"1afacc90-2b81-42c2-86af-1622afdfca74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5da0f1-8q8fm" Nov 02 23:59:49.191000 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:49.190978 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1afacc90-2b81-42c2-86af-1622afdfca74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f5da0f1-8q8fm\" (UID: \"1afacc90-2b81-42c2-86af-1622afdfca74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5da0f1-8q8fm" Nov 02 23:59:49.192996 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:49.192973 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1afacc90-2b81-42c2-86af-1622afdfca74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f5da0f1-8q8fm\" (UID: \"1afacc90-2b81-42c2-86af-1622afdfca74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5da0f1-8q8fm" Nov 02 23:59:49.252045 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:49.252011 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hln88\" (UniqueName: \"kubernetes.io/projected/1afacc90-2b81-42c2-86af-1622afdfca74-kube-api-access-hln88\") pod \"maas-api-revoke-sub-7f5da0f1-8q8fm\" (UID: \"1afacc90-2b81-42c2-86af-1622afdfca74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5da0f1-8q8fm" Nov 02 23:59:49.381873 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:49.381785 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5da0f1-8q8fm" Nov 02 23:59:49.761041 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:49.761016 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f5da0f1-8q8fm"] Nov 02 23:59:49.763080 ip-10-0-135-250 kubenswrapper[5114]: W1102 23:59:49.763051 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1afacc90_2b81_42c2_86af_1622afdfca74.slice/crio-c1d18b856041d46a21df964a820d97c4a6ce4a212fc4122750c020f044c0aa5e WatchSource:0}: Error finding container c1d18b856041d46a21df964a820d97c4a6ce4a212fc4122750c020f044c0aa5e: Status 404 returned error can't find the container with id c1d18b856041d46a21df964a820d97c4a6ce4a212fc4122750c020f044c0aa5e Nov 02 23:59:50.238656 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:50.238617 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5da0f1-8q8fm" event={"ID":"1afacc90-2b81-42c2-86af-1622afdfca74","Type":"ContainerStarted","Data":"dee311b64354a4dfbbdec9b7b352dc86ba0a98b705ab864e2caca6e27b3326b9"} Nov 02 23:59:50.239108 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:50.238671 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5da0f1-8q8fm" event={"ID":"1afacc90-2b81-42c2-86af-1622afdfca74","Type":"ContainerStarted","Data":"c1d18b856041d46a21df964a820d97c4a6ce4a212fc4122750c020f044c0aa5e"} Nov 02 23:59:50.353521 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:50.353438 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5da0f1-8q8fm" podStartSLOduration=1.35342195 podStartE2EDuration="1.35342195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:59:50.351987969 +0000 UTC m=+4265.705021344" watchObservedRunningTime="2026-11-02 23:59:50.35342195 +0000 UTC m=+4265.706455325" Nov 02 23:59:51.246535 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:51.246477 5114 generic.go:356] "Generic (PLEG): container finished" podID="1afacc90-2b81-42c2-86af-1622afdfca74" containerID="dee311b64354a4dfbbdec9b7b352dc86ba0a98b705ab864e2caca6e27b3326b9" exitCode=0 Nov 02 23:59:51.246535 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:51.246529 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5da0f1-8q8fm" event={"ID":"1afacc90-2b81-42c2-86af-1622afdfca74","Type":"ContainerDied","Data":"dee311b64354a4dfbbdec9b7b352dc86ba0a98b705ab864e2caca6e27b3326b9"} Nov 02 23:59:52.394293 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:52.394261 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5da0f1-8q8fm" Nov 02 23:59:52.430897 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:52.430860 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1afacc90-2b81-42c2-86af-1622afdfca74-kube-api-access-hln88\" (UniqueName: \"kubernetes.io/projected/1afacc90-2b81-42c2-86af-1622afdfca74-kube-api-access-hln88\") pod \"1afacc90-2b81-42c2-86af-1622afdfca74\" (UID: \"1afacc90-2b81-42c2-86af-1622afdfca74\") " Nov 02 23:59:52.431097 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:52.430981 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1afacc90-2b81-42c2-86af-1622afdfca74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1afacc90-2b81-42c2-86af-1622afdfca74-maas-api-openshift-ca\") pod \"1afacc90-2b81-42c2-86af-1622afdfca74\" (UID: \"1afacc90-2b81-42c2-86af-1622afdfca74\") " Nov 02 23:59:52.431097 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:52.431024 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1afacc90-2b81-42c2-86af-1622afdfca74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1afacc90-2b81-42c2-86af-1622afdfca74-maas-api-xks-ca\") pod \"1afacc90-2b81-42c2-86af-1622afdfca74\" (UID: \"1afacc90-2b81-42c2-86af-1622afdfca74\") " Nov 02 23:59:52.431387 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:52.431364 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1afacc90-2b81-42c2-86af-1622afdfca74-maas-api-openshift-ca" pod "1afacc90-2b81-42c2-86af-1622afdfca74" (UID: "1afacc90-2b81-42c2-86af-1622afdfca74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:59:52.433143 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:52.433120 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1afacc90-2b81-42c2-86af-1622afdfca74-kube-api-access-hln88" pod "1afacc90-2b81-42c2-86af-1622afdfca74" (UID: "1afacc90-2b81-42c2-86af-1622afdfca74"). InnerVolumeSpecName "kube-api-access-hln88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:59:52.433277 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:52.433257 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1afacc90-2b81-42c2-86af-1622afdfca74-maas-api-xks-ca" pod "1afacc90-2b81-42c2-86af-1622afdfca74" (UID: "1afacc90-2b81-42c2-86af-1622afdfca74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:59:52.532446 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:52.532371 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1afacc90-2b81-42c2-86af-1622afdfca74-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:52.532446 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:52.532403 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1afacc90-2b81-42c2-86af-1622afdfca74-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:52.532446 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:52.532414 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hln88\" (UniqueName: \"kubernetes.io/projected/1afacc90-2b81-42c2-86af-1622afdfca74-kube-api-access-hln88\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:53.259819 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:53.259720 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5da0f1-8q8fm" event={"ID":"1afacc90-2b81-42c2-86af-1622afdfca74","Type":"ContainerDied","Data":"c1d18b856041d46a21df964a820d97c4a6ce4a212fc4122750c020f044c0aa5e"} Nov 02 23:59:53.259819 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:53.259760 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5da0f1-8q8fm" Nov 02 23:59:53.259819 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:53.259784 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1d18b856041d46a21df964a820d97c4a6ce4a212fc4122750c020f044c0aa5e" Nov 02 23:59:58.048412 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:58.048368 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8972c560-7vfvd"] Nov 02 23:59:58.098075 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:58.098036 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8972c560-7vfvd"] Nov 02 23:59:58.793163 ip-10-0-135-250 kubenswrapper[5114]: I1102 23:59:58.793126 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4b518d0-6faf-4eaf-b824-9d37ea5870f4" path="/var/lib/kubelet/pods/f4b518d0-6faf-4eaf-b824-9d37ea5870f4/volumes" Nov 03 00:00:00.175696 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.175653 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs"] Nov 03 00:00:00.176128 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.176114 5114 cpu_manager.go:414] "containerMap: removing container" podUID="1afacc90-2b81-42c2-86af-1622afdfca74" containerName="revoke-keys" Nov 03 00:00:00.176173 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.176129 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1afacc90-2b81-42c2-86af-1622afdfca74" containerName="revoke-keys" Nov 03 00:00:00.176220 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.176211 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ab21771-e42c-462f-b83f-ee47948059dd" containerName="istio-proxy" Nov 03 00:00:00.176256 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.176225 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="1afacc90-2b81-42c2-86af-1622afdfca74" containerName="revoke-keys" Nov 03 00:00:00.180692 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.180668 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs" Nov 03 00:00:00.182163 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.182138 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw"] Nov 03 00:00:00.186383 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.186357 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n"] Nov 03 00:00:00.186580 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.186543 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw" Nov 03 00:00:00.190260 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.190236 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n" Nov 03 00:00:00.226935 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.226899 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx"] Nov 03 00:00:00.231543 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.231516 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx" Nov 03 00:00:00.231887 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.231866 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m"] Nov 03 00:00:00.235543 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.235518 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf"] Nov 03 00:00:00.235683 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.235588 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m" Nov 03 00:00:00.239219 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.239195 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf" Nov 03 00:00:00.272423 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.272392 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 03 00:00:00.275271 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.275240 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd"] Nov 03 00:00:00.279085 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.279058 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd" Nov 03 00:00:00.291124 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.291083 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs"] Nov 03 00:00:00.293620 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.293592 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n"] Nov 03 00:00:00.296553 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.296527 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw"] Nov 03 00:00:00.338609 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.337865 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89ca75ec-2516-4f65-b600-4b6d9c72908d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf\" (UID: \"89ca75ec-2516-4f65-b600-4b6d9c72908d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf" Nov 03 00:00:00.338609 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.337917 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b68115-0b68-4d74-96ec-e70f18391969-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw\" (UID: \"53b68115-0b68-4d74-96ec-e70f18391969\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw" Nov 03 00:00:00.338609 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.337947 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfxjz\" (UniqueName: \"kubernetes.io/projected/ccf010d7-6b97-4824-83da-9acac85d87bd-kube-api-access-cfxjz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx\" (UID: \"ccf010d7-6b97-4824-83da-9acac85d87bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx" Nov 03 00:00:00.338609 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.337985 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbf6b\" (UniqueName: \"kubernetes.io/projected/730f6d5e-c29d-491d-b40d-197ee83b76ff-kube-api-access-rbf6b\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n\" (UID: \"730f6d5e-c29d-491d-b40d-197ee83b76ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n" Nov 03 00:00:00.338609 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.338060 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/730f6d5e-c29d-491d-b40d-197ee83b76ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n\" (UID: \"730f6d5e-c29d-491d-b40d-197ee83b76ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n" Nov 03 00:00:00.338609 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.338102 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a294a5e-b315-4c9e-82f3-47f5dd2d729e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs\" (UID: \"1a294a5e-b315-4c9e-82f3-47f5dd2d729e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs" Nov 03 00:00:00.338609 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.338126 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccf010d7-6b97-4824-83da-9acac85d87bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx\" (UID: \"ccf010d7-6b97-4824-83da-9acac85d87bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx" Nov 03 00:00:00.338609 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.338165 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28cx2\" (UniqueName: \"kubernetes.io/projected/1a294a5e-b315-4c9e-82f3-47f5dd2d729e-kube-api-access-28cx2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs\" (UID: \"1a294a5e-b315-4c9e-82f3-47f5dd2d729e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs" Nov 03 00:00:00.338609 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.338220 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnq5r\" (UniqueName: \"kubernetes.io/projected/53b68115-0b68-4d74-96ec-e70f18391969-kube-api-access-hnq5r\") pod \"maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw\" (UID: \"53b68115-0b68-4d74-96ec-e70f18391969\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw" Nov 03 00:00:00.338609 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.338247 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg44s\" (UniqueName: \"kubernetes.io/projected/5762c029-f662-41b5-9941-86af30749b7d-kube-api-access-hg44s\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m\" (UID: \"5762c029-f662-41b5-9941-86af30749b7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m" Nov 03 00:00:00.338609 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.338377 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kk5q\" (UniqueName: \"kubernetes.io/projected/89ca75ec-2516-4f65-b600-4b6d9c72908d-kube-api-access-4kk5q\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf\" (UID: \"89ca75ec-2516-4f65-b600-4b6d9c72908d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf" Nov 03 00:00:00.338609 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.338439 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5762c029-f662-41b5-9941-86af30749b7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m\" (UID: \"5762c029-f662-41b5-9941-86af30749b7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m" Nov 03 00:00:00.346644 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.343254 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx"] Nov 03 00:00:00.396024 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.395629 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf"] Nov 03 00:00:00.399288 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.399242 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd"] Nov 03 00:00:00.439208 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.439115 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/730f6d5e-c29d-491d-b40d-197ee83b76ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n\" (UID: \"730f6d5e-c29d-491d-b40d-197ee83b76ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n" Nov 03 00:00:00.439208 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.439158 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a294a5e-b315-4c9e-82f3-47f5dd2d729e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs\" (UID: \"1a294a5e-b315-4c9e-82f3-47f5dd2d729e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs" Nov 03 00:00:00.439208 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.439178 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccf010d7-6b97-4824-83da-9acac85d87bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx\" (UID: \"ccf010d7-6b97-4824-83da-9acac85d87bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx" Nov 03 00:00:00.439208 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.439202 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28cx2\" (UniqueName: \"kubernetes.io/projected/1a294a5e-b315-4c9e-82f3-47f5dd2d729e-kube-api-access-28cx2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs\" (UID: \"1a294a5e-b315-4c9e-82f3-47f5dd2d729e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs" Nov 03 00:00:00.439589 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.439249 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnq5r\" (UniqueName: \"kubernetes.io/projected/53b68115-0b68-4d74-96ec-e70f18391969-kube-api-access-hnq5r\") pod \"maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw\" (UID: \"53b68115-0b68-4d74-96ec-e70f18391969\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw" Nov 03 00:00:00.439589 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.439279 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg44s\" (UniqueName: \"kubernetes.io/projected/5762c029-f662-41b5-9941-86af30749b7d-kube-api-access-hg44s\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m\" (UID: \"5762c029-f662-41b5-9941-86af30749b7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m" Nov 03 00:00:00.439589 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.439386 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kk5q\" (UniqueName: \"kubernetes.io/projected/89ca75ec-2516-4f65-b600-4b6d9c72908d-kube-api-access-4kk5q\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf\" (UID: \"89ca75ec-2516-4f65-b600-4b6d9c72908d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf" Nov 03 00:00:00.439589 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.439438 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhrjb\" (UniqueName: \"kubernetes.io/projected/2da46d94-2113-49ed-8349-d0a0f38cde59-kube-api-access-mhrjb\") pod \"maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd\" (UID: \"2da46d94-2113-49ed-8349-d0a0f38cde59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd" Nov 03 00:00:00.439589 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.439503 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5762c029-f662-41b5-9941-86af30749b7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m\" (UID: \"5762c029-f662-41b5-9941-86af30749b7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m" Nov 03 00:00:00.439589 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.439552 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89ca75ec-2516-4f65-b600-4b6d9c72908d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf\" (UID: \"89ca75ec-2516-4f65-b600-4b6d9c72908d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf" Nov 03 00:00:00.439589 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.439583 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b68115-0b68-4d74-96ec-e70f18391969-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw\" (UID: \"53b68115-0b68-4d74-96ec-e70f18391969\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw" Nov 03 00:00:00.439962 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.439608 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfxjz\" (UniqueName: \"kubernetes.io/projected/ccf010d7-6b97-4824-83da-9acac85d87bd-kube-api-access-cfxjz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx\" (UID: \"ccf010d7-6b97-4824-83da-9acac85d87bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx" Nov 03 00:00:00.439962 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.439644 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbf6b\" (UniqueName: \"kubernetes.io/projected/730f6d5e-c29d-491d-b40d-197ee83b76ff-kube-api-access-rbf6b\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n\" (UID: \"730f6d5e-c29d-491d-b40d-197ee83b76ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n" Nov 03 00:00:00.439962 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.439688 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da46d94-2113-49ed-8349-d0a0f38cde59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd\" (UID: \"2da46d94-2113-49ed-8349-d0a0f38cde59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd" Nov 03 00:00:00.440139 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.440119 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccf010d7-6b97-4824-83da-9acac85d87bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx\" (UID: \"ccf010d7-6b97-4824-83da-9acac85d87bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx" Nov 03 00:00:00.440198 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.440117 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a294a5e-b315-4c9e-82f3-47f5dd2d729e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs\" (UID: \"1a294a5e-b315-4c9e-82f3-47f5dd2d729e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs" Nov 03 00:00:00.440198 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.440123 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/730f6d5e-c29d-491d-b40d-197ee83b76ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n\" (UID: \"730f6d5e-c29d-491d-b40d-197ee83b76ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n" Nov 03 00:00:00.440889 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.440861 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5762c029-f662-41b5-9941-86af30749b7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m\" (UID: \"5762c029-f662-41b5-9941-86af30749b7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m" Nov 03 00:00:00.441081 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.440892 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b68115-0b68-4d74-96ec-e70f18391969-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw\" (UID: \"53b68115-0b68-4d74-96ec-e70f18391969\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw" Nov 03 00:00:00.441081 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.440945 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89ca75ec-2516-4f65-b600-4b6d9c72908d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf\" (UID: \"89ca75ec-2516-4f65-b600-4b6d9c72908d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf" Nov 03 00:00:00.502229 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.501994 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m"] Nov 03 00:00:00.502824 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.502794 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbf6b\" (UniqueName: \"kubernetes.io/projected/730f6d5e-c29d-491d-b40d-197ee83b76ff-kube-api-access-rbf6b\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n\" (UID: \"730f6d5e-c29d-491d-b40d-197ee83b76ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n" Nov 03 00:00:00.502979 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.502792 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg44s\" (UniqueName: \"kubernetes.io/projected/5762c029-f662-41b5-9941-86af30749b7d-kube-api-access-hg44s\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m\" (UID: \"5762c029-f662-41b5-9941-86af30749b7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m" Nov 03 00:00:00.503925 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.503865 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28cx2\" (UniqueName: \"kubernetes.io/projected/1a294a5e-b315-4c9e-82f3-47f5dd2d729e-kube-api-access-28cx2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs\" (UID: \"1a294a5e-b315-4c9e-82f3-47f5dd2d729e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs" Nov 03 00:00:00.505168 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.505141 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfxjz\" (UniqueName: \"kubernetes.io/projected/ccf010d7-6b97-4824-83da-9acac85d87bd-kube-api-access-cfxjz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx\" (UID: \"ccf010d7-6b97-4824-83da-9acac85d87bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx" Nov 03 00:00:00.505707 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.505683 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kk5q\" (UniqueName: \"kubernetes.io/projected/89ca75ec-2516-4f65-b600-4b6d9c72908d-kube-api-access-4kk5q\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf\" (UID: \"89ca75ec-2516-4f65-b600-4b6d9c72908d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf" Nov 03 00:00:00.507140 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.507113 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnq5r\" (UniqueName: \"kubernetes.io/projected/53b68115-0b68-4d74-96ec-e70f18391969-kube-api-access-hnq5r\") pod \"maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw\" (UID: \"53b68115-0b68-4d74-96ec-e70f18391969\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw" Nov 03 00:00:00.522598 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.522564 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs" Nov 03 00:00:00.528087 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.528061 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw" Nov 03 00:00:00.536017 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.535948 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n" Nov 03 00:00:00.540935 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.540905 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhrjb\" (UniqueName: \"kubernetes.io/projected/2da46d94-2113-49ed-8349-d0a0f38cde59-kube-api-access-mhrjb\") pod \"maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd\" (UID: \"2da46d94-2113-49ed-8349-d0a0f38cde59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd" Nov 03 00:00:00.541092 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.541014 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da46d94-2113-49ed-8349-d0a0f38cde59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd\" (UID: \"2da46d94-2113-49ed-8349-d0a0f38cde59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd" Nov 03 00:00:00.542457 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.542426 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da46d94-2113-49ed-8349-d0a0f38cde59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd\" (UID: \"2da46d94-2113-49ed-8349-d0a0f38cde59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd" Nov 03 00:00:00.545986 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.545944 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx" Nov 03 00:00:00.554276 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.554173 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m" Nov 03 00:00:00.567458 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.567430 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf" Nov 03 00:00:00.624733 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.624689 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhrjb\" (UniqueName: \"kubernetes.io/projected/2da46d94-2113-49ed-8349-d0a0f38cde59-kube-api-access-mhrjb\") pod \"maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd\" (UID: \"2da46d94-2113-49ed-8349-d0a0f38cde59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd" Nov 03 00:00:00.894746 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.894709 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd" Nov 03 00:00:00.937639 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.937612 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs"] Nov 03 00:00:00.941147 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:00:00.941112 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a294a5e_b315_4c9e_82f3_47f5dd2d729e.slice/crio-66dd1b38ed9c83d7f1cbae8ee84e83e507e00d5e38f5bd8726f0d07fcc38bb38 WatchSource:0}: Error finding container 66dd1b38ed9c83d7f1cbae8ee84e83e507e00d5e38f5bd8726f0d07fcc38bb38: Status 404 returned error can't find the container with id 66dd1b38ed9c83d7f1cbae8ee84e83e507e00d5e38f5bd8726f0d07fcc38bb38 Nov 03 00:00:00.987895 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:00.987860 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m"] Nov 03 00:00:00.989644 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:00:00.989613 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5762c029_f662_41b5_9941_86af30749b7d.slice/crio-5709d8595a8718474557b9ed0bc68fdb110bba58b3fdc97c3836a2b9d5440a74 WatchSource:0}: Error finding container 5709d8595a8718474557b9ed0bc68fdb110bba58b3fdc97c3836a2b9d5440a74: Status 404 returned error can't find the container with id 5709d8595a8718474557b9ed0bc68fdb110bba58b3fdc97c3836a2b9d5440a74 Nov 03 00:00:01.037496 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:01.037438 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx"] Nov 03 00:00:01.040678 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:01.040565 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n"] Nov 03 00:00:01.049121 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:01.049090 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf"] Nov 03 00:00:01.058539 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:00:01.058506 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ca75ec_2516_4f65_b600_4b6d9c72908d.slice/crio-849a699d8b8d996acef6d7c479e6671ceaed4877caba35da3fa1f2f82a54ca03 WatchSource:0}: Error finding container 849a699d8b8d996acef6d7c479e6671ceaed4877caba35da3fa1f2f82a54ca03: Status 404 returned error can't find the container with id 849a699d8b8d996acef6d7c479e6671ceaed4877caba35da3fa1f2f82a54ca03 Nov 03 00:00:01.142171 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:01.142141 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw"] Nov 03 00:00:01.224556 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:00:01.224510 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b68115_0b68_4d74_96ec_e70f18391969.slice/crio-d8c764ae72181dff1cb6b0f6f28ebacf59fce1661eae9af20df792be790b4ac5 WatchSource:0}: Error finding container d8c764ae72181dff1cb6b0f6f28ebacf59fce1661eae9af20df792be790b4ac5: Status 404 returned error can't find the container with id d8c764ae72181dff1cb6b0f6f28ebacf59fce1661eae9af20df792be790b4ac5 Nov 03 00:00:01.277812 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:01.277784 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd"] Nov 03 00:00:01.279843 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:00:01.279817 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da46d94_2113_49ed_8349_d0a0f38cde59.slice/crio-980d1a55367161e586686d65227c59d3bb9d71005e39d9c6e22702a19b95b6de WatchSource:0}: Error finding container 980d1a55367161e586686d65227c59d3bb9d71005e39d9c6e22702a19b95b6de: Status 404 returned error can't find the container with id 980d1a55367161e586686d65227c59d3bb9d71005e39d9c6e22702a19b95b6de Nov 03 00:00:01.308078 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:01.308043 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m" event={"ID":"5762c029-f662-41b5-9941-86af30749b7d","Type":"ContainerStarted","Data":"5709d8595a8718474557b9ed0bc68fdb110bba58b3fdc97c3836a2b9d5440a74"} Nov 03 00:00:01.312809 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:01.310063 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf" event={"ID":"89ca75ec-2516-4f65-b600-4b6d9c72908d","Type":"ContainerStarted","Data":"849a699d8b8d996acef6d7c479e6671ceaed4877caba35da3fa1f2f82a54ca03"} Nov 03 00:00:01.312809 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:01.311594 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw" event={"ID":"53b68115-0b68-4d74-96ec-e70f18391969","Type":"ContainerStarted","Data":"d8c764ae72181dff1cb6b0f6f28ebacf59fce1661eae9af20df792be790b4ac5"} Nov 03 00:00:01.313361 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:01.313337 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx" event={"ID":"ccf010d7-6b97-4824-83da-9acac85d87bd","Type":"ContainerStarted","Data":"517e9050dce4d4783499299adf1ca9d2a4c21676c2eeb1c0146b64f4b2453132"} Nov 03 00:00:01.314600 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:01.314580 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n" event={"ID":"730f6d5e-c29d-491d-b40d-197ee83b76ff","Type":"ContainerStarted","Data":"0ba66d28e061ef8541c9a11d124246b09c12abe80efbf775c66ede49ea18fd76"} Nov 03 00:00:01.315814 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:01.315791 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd" event={"ID":"2da46d94-2113-49ed-8349-d0a0f38cde59","Type":"ContainerStarted","Data":"980d1a55367161e586686d65227c59d3bb9d71005e39d9c6e22702a19b95b6de"} Nov 03 00:00:01.316892 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:01.316869 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs" event={"ID":"1a294a5e-b315-4c9e-82f3-47f5dd2d729e","Type":"ContainerStarted","Data":"66dd1b38ed9c83d7f1cbae8ee84e83e507e00d5e38f5bd8726f0d07fcc38bb38"} Nov 03 00:00:02.323900 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:02.323868 5114 generic.go:356] "Generic (PLEG): container finished" podID="1a294a5e-b315-4c9e-82f3-47f5dd2d729e" containerID="efd07f6dd54e11410ce2dc4bf5a9e0d5636d085694eee58bbc346eb76b498fdd" exitCode=60 Nov 03 00:00:02.324338 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:02.323955 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs" event={"ID":"1a294a5e-b315-4c9e-82f3-47f5dd2d729e","Type":"ContainerDied","Data":"efd07f6dd54e11410ce2dc4bf5a9e0d5636d085694eee58bbc346eb76b498fdd"} Nov 03 00:00:02.324338 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:02.324218 5114 scope.go:122] "RemoveContainer" containerID="efd07f6dd54e11410ce2dc4bf5a9e0d5636d085694eee58bbc346eb76b498fdd" Nov 03 00:00:02.325656 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:02.325629 5114 generic.go:356] "Generic (PLEG): container finished" podID="5762c029-f662-41b5-9941-86af30749b7d" containerID="6bf51b13cc66797285c36260d3e1f9167e75676a6a50e5d612178d4197bb9604" exitCode=60 Nov 03 00:00:02.325793 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:02.325698 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m" event={"ID":"5762c029-f662-41b5-9941-86af30749b7d","Type":"ContainerDied","Data":"6bf51b13cc66797285c36260d3e1f9167e75676a6a50e5d612178d4197bb9604"} Nov 03 00:00:02.325982 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:02.325966 5114 scope.go:122] "RemoveContainer" containerID="6bf51b13cc66797285c36260d3e1f9167e75676a6a50e5d612178d4197bb9604" Nov 03 00:00:02.327535 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:02.327417 5114 generic.go:356] "Generic (PLEG): container finished" podID="89ca75ec-2516-4f65-b600-4b6d9c72908d" containerID="dd3505186bb738fcb1c0c83eb28d8723467e7308c4d5e7d0d58d5d3e9ded6a48" exitCode=60 Nov 03 00:00:02.327535 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:02.327510 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf" event={"ID":"89ca75ec-2516-4f65-b600-4b6d9c72908d","Type":"ContainerDied","Data":"dd3505186bb738fcb1c0c83eb28d8723467e7308c4d5e7d0d58d5d3e9ded6a48"} Nov 03 00:00:02.327739 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:02.327722 5114 scope.go:122] "RemoveContainer" containerID="dd3505186bb738fcb1c0c83eb28d8723467e7308c4d5e7d0d58d5d3e9ded6a48" Nov 03 00:00:02.329078 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:02.329057 5114 generic.go:356] "Generic (PLEG): container finished" podID="53b68115-0b68-4d74-96ec-e70f18391969" containerID="0f2f578379e54c3e4c4d5c7f898b9b709f35e570da216319c1fb8e7cc3b358b1" exitCode=60 Nov 03 00:00:02.329169 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:02.329144 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw" event={"ID":"53b68115-0b68-4d74-96ec-e70f18391969","Type":"ContainerDied","Data":"0f2f578379e54c3e4c4d5c7f898b9b709f35e570da216319c1fb8e7cc3b358b1"} Nov 03 00:00:02.329321 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:02.329309 5114 scope.go:122] "RemoveContainer" containerID="0f2f578379e54c3e4c4d5c7f898b9b709f35e570da216319c1fb8e7cc3b358b1" Nov 03 00:00:02.330714 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:02.330694 5114 generic.go:356] "Generic (PLEG): container finished" podID="ccf010d7-6b97-4824-83da-9acac85d87bd" containerID="292ad08dd9d4545136efa1822b8225e9754f76a68ed045f095d87ced6245a573" exitCode=60 Nov 03 00:00:02.330816 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:02.330768 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx" event={"ID":"ccf010d7-6b97-4824-83da-9acac85d87bd","Type":"ContainerDied","Data":"292ad08dd9d4545136efa1822b8225e9754f76a68ed045f095d87ced6245a573"} Nov 03 00:00:02.330971 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:02.330954 5114 scope.go:122] "RemoveContainer" containerID="292ad08dd9d4545136efa1822b8225e9754f76a68ed045f095d87ced6245a573" Nov 03 00:00:02.332444 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:02.332422 5114 generic.go:356] "Generic (PLEG): container finished" podID="730f6d5e-c29d-491d-b40d-197ee83b76ff" containerID="407e555b3046a999de96b9071f4792e15e6c2ccd47f7717488482459d17edb78" exitCode=60 Nov 03 00:00:02.332615 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:02.332507 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n" event={"ID":"730f6d5e-c29d-491d-b40d-197ee83b76ff","Type":"ContainerDied","Data":"407e555b3046a999de96b9071f4792e15e6c2ccd47f7717488482459d17edb78"} Nov 03 00:00:02.332731 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:02.332713 5114 scope.go:122] "RemoveContainer" containerID="407e555b3046a999de96b9071f4792e15e6c2ccd47f7717488482459d17edb78" Nov 03 00:00:02.334859 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:02.334835 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd" event={"ID":"2da46d94-2113-49ed-8349-d0a0f38cde59","Type":"ContainerStarted","Data":"4128e57631f02a1ac2bacd9c6a084b3bef36eb29fe0e36f9510caeacd337703d"} Nov 03 00:00:02.782200 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:02.781961 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd" podStartSLOduration=2.781941164 podStartE2EDuration="2.781941164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:02.781520424 +0000 UTC m=+4278.134553800" watchObservedRunningTime="2026-11-03 00:00:02.781941164 +0000 UTC m=+4278.134974541" Nov 03 00:00:03.343788 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:03.343748 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m" event={"ID":"5762c029-f662-41b5-9941-86af30749b7d","Type":"ContainerStarted","Data":"b7a800731fd9ac2ebcb87dff06052079dc960bea279ce1c61ffca2f71027fbc6"} Nov 03 00:00:03.345824 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:03.345792 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf" event={"ID":"89ca75ec-2516-4f65-b600-4b6d9c72908d","Type":"ContainerStarted","Data":"0501005e23821c03c16aa70b49981ba0c71f97b35de7cd78e1d613ec934d60f9"} Nov 03 00:00:03.347647 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:03.347625 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw" event={"ID":"53b68115-0b68-4d74-96ec-e70f18391969","Type":"ContainerStarted","Data":"c291c2fdce700f77b69f57400c1c61c3cd7c9b8a3832b0b1c02c898a528c8e7b"} Nov 03 00:00:03.349556 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:03.349536 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx" event={"ID":"ccf010d7-6b97-4824-83da-9acac85d87bd","Type":"ContainerStarted","Data":"12151789d18a774582344f6578b73d0fb47d727b79fc3d750a721cce9819d67d"} Nov 03 00:00:03.351389 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:03.351362 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n" event={"ID":"730f6d5e-c29d-491d-b40d-197ee83b76ff","Type":"ContainerStarted","Data":"a0936bf0fde8fde82002cc02263f3a349ff2a97466633e4008d33678c7f57eb2"} Nov 03 00:00:03.352827 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:03.352806 5114 generic.go:356] "Generic (PLEG): container finished" podID="2da46d94-2113-49ed-8349-d0a0f38cde59" containerID="4128e57631f02a1ac2bacd9c6a084b3bef36eb29fe0e36f9510caeacd337703d" exitCode=60 Nov 03 00:00:03.352949 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:03.352891 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd" event={"ID":"2da46d94-2113-49ed-8349-d0a0f38cde59","Type":"ContainerDied","Data":"4128e57631f02a1ac2bacd9c6a084b3bef36eb29fe0e36f9510caeacd337703d"} Nov 03 00:00:03.353048 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:03.353030 5114 scope.go:122] "RemoveContainer" containerID="4128e57631f02a1ac2bacd9c6a084b3bef36eb29fe0e36f9510caeacd337703d" Nov 03 00:00:03.354882 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:03.354865 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs" event={"ID":"1a294a5e-b315-4c9e-82f3-47f5dd2d729e","Type":"ContainerStarted","Data":"79a79e67b543fd4145aa34a8b4a4f38b6f96df118e832eedfacd940963da9b19"} Nov 03 00:00:03.458852 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:03.458788 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m" podStartSLOduration=3.458771636 podStartE2EDuration="3.458771636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:03.457793638 +0000 UTC m=+4278.810827013" watchObservedRunningTime="2026-11-03 00:00:03.458771636 +0000 UTC m=+4278.811805048" Nov 03 00:00:03.573388 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:03.573021 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs" podStartSLOduration=3.57300211 podStartE2EDuration="3.57300211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:03.571853917 +0000 UTC m=+4278.924887293" watchObservedRunningTime="2026-11-03 00:00:03.57300211 +0000 UTC m=+4278.926035486" Nov 03 00:00:03.686617 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:03.686560 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx" podStartSLOduration=3.686540789 podStartE2EDuration="3.686540789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:03.685687818 +0000 UTC m=+4279.038721194" watchObservedRunningTime="2026-11-03 00:00:03.686540789 +0000 UTC m=+4279.039574164" Nov 03 00:00:03.916990 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:03.916933 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n" podStartSLOduration=3.91691814 podStartE2EDuration="3.91691814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:03.915612469 +0000 UTC m=+4279.268645845" watchObservedRunningTime="2026-11-03 00:00:03.91691814 +0000 UTC m=+4279.269951515" Nov 03 00:00:04.031767 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.031708 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf" podStartSLOduration=4.031694755 podStartE2EDuration="4.031694755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:04.029414125 +0000 UTC m=+4279.382447501" watchObservedRunningTime="2026-11-03 00:00:04.031694755 +0000 UTC m=+4279.384728130" Nov 03 00:00:04.145455 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.145402 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw" podStartSLOduration=4.145389769 podStartE2EDuration="4.145389769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:04.145235796 +0000 UTC m=+4279.498269170" watchObservedRunningTime="2026-11-03 00:00:04.145389769 +0000 UTC m=+4279.498423144" Nov 03 00:00:04.362970 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.362932 5114 generic.go:356] "Generic (PLEG): container finished" podID="5762c029-f662-41b5-9941-86af30749b7d" containerID="b7a800731fd9ac2ebcb87dff06052079dc960bea279ce1c61ffca2f71027fbc6" exitCode=60 Nov 03 00:00:04.363471 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.363011 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m" event={"ID":"5762c029-f662-41b5-9941-86af30749b7d","Type":"ContainerDied","Data":"b7a800731fd9ac2ebcb87dff06052079dc960bea279ce1c61ffca2f71027fbc6"} Nov 03 00:00:04.363471 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.363075 5114 scope.go:122] "RemoveContainer" containerID="6bf51b13cc66797285c36260d3e1f9167e75676a6a50e5d612178d4197bb9604" Nov 03 00:00:04.363471 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.363355 5114 scope.go:122] "RemoveContainer" containerID="b7a800731fd9ac2ebcb87dff06052079dc960bea279ce1c61ffca2f71027fbc6" Nov 03 00:00:04.363743 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:00:04.363717 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m_odh-ai-gateway-infra(5762c029-f662-41b5-9941-86af30749b7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m" podUID="5762c029-f662-41b5-9941-86af30749b7d" Nov 03 00:00:04.365247 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.365225 5114 generic.go:356] "Generic (PLEG): container finished" podID="89ca75ec-2516-4f65-b600-4b6d9c72908d" containerID="0501005e23821c03c16aa70b49981ba0c71f97b35de7cd78e1d613ec934d60f9" exitCode=60 Nov 03 00:00:04.365347 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.365268 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf" event={"ID":"89ca75ec-2516-4f65-b600-4b6d9c72908d","Type":"ContainerDied","Data":"0501005e23821c03c16aa70b49981ba0c71f97b35de7cd78e1d613ec934d60f9"} Nov 03 00:00:04.365636 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.365618 5114 scope.go:122] "RemoveContainer" containerID="0501005e23821c03c16aa70b49981ba0c71f97b35de7cd78e1d613ec934d60f9" Nov 03 00:00:04.365937 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:00:04.365910 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf_odh-ai-gateway-infra(89ca75ec-2516-4f65-b600-4b6d9c72908d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf" podUID="89ca75ec-2516-4f65-b600-4b6d9c72908d" Nov 03 00:00:04.367308 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.367285 5114 generic.go:356] "Generic (PLEG): container finished" podID="53b68115-0b68-4d74-96ec-e70f18391969" containerID="c291c2fdce700f77b69f57400c1c61c3cd7c9b8a3832b0b1c02c898a528c8e7b" exitCode=60 Nov 03 00:00:04.367427 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.367378 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw" event={"ID":"53b68115-0b68-4d74-96ec-e70f18391969","Type":"ContainerDied","Data":"c291c2fdce700f77b69f57400c1c61c3cd7c9b8a3832b0b1c02c898a528c8e7b"} Nov 03 00:00:04.367717 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.367697 5114 scope.go:122] "RemoveContainer" containerID="c291c2fdce700f77b69f57400c1c61c3cd7c9b8a3832b0b1c02c898a528c8e7b" Nov 03 00:00:04.368017 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:00:04.367998 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw_odh-ai-gateway-infra(53b68115-0b68-4d74-96ec-e70f18391969)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw" podUID="53b68115-0b68-4d74-96ec-e70f18391969" Nov 03 00:00:04.369411 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.369387 5114 generic.go:356] "Generic (PLEG): container finished" podID="ccf010d7-6b97-4824-83da-9acac85d87bd" containerID="12151789d18a774582344f6578b73d0fb47d727b79fc3d750a721cce9819d67d" exitCode=60 Nov 03 00:00:04.369555 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.369464 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx" event={"ID":"ccf010d7-6b97-4824-83da-9acac85d87bd","Type":"ContainerDied","Data":"12151789d18a774582344f6578b73d0fb47d727b79fc3d750a721cce9819d67d"} Nov 03 00:00:04.369754 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.369738 5114 scope.go:122] "RemoveContainer" containerID="12151789d18a774582344f6578b73d0fb47d727b79fc3d750a721cce9819d67d" Nov 03 00:00:04.370019 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:00:04.369996 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx_odh-ai-gateway-infra(ccf010d7-6b97-4824-83da-9acac85d87bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx" podUID="ccf010d7-6b97-4824-83da-9acac85d87bd" Nov 03 00:00:04.371776 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.371751 5114 generic.go:356] "Generic (PLEG): container finished" podID="730f6d5e-c29d-491d-b40d-197ee83b76ff" containerID="a0936bf0fde8fde82002cc02263f3a349ff2a97466633e4008d33678c7f57eb2" exitCode=60 Nov 03 00:00:04.371898 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.371847 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n" event={"ID":"730f6d5e-c29d-491d-b40d-197ee83b76ff","Type":"ContainerDied","Data":"a0936bf0fde8fde82002cc02263f3a349ff2a97466633e4008d33678c7f57eb2"} Nov 03 00:00:04.372071 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.372056 5114 scope.go:122] "RemoveContainer" containerID="a0936bf0fde8fde82002cc02263f3a349ff2a97466633e4008d33678c7f57eb2" Nov 03 00:00:04.372317 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:00:04.372295 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n_odh-ai-gateway-infra(730f6d5e-c29d-491d-b40d-197ee83b76ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n" podUID="730f6d5e-c29d-491d-b40d-197ee83b76ff" Nov 03 00:00:04.373892 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.373868 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd" event={"ID":"2da46d94-2113-49ed-8349-d0a0f38cde59","Type":"ContainerStarted","Data":"e212400dbe0062fbdb35a7f3475db30dbc7390386b3f9cfe6f787bcfb99bcdba"} Nov 03 00:00:04.375658 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.375642 5114 generic.go:356] "Generic (PLEG): container finished" podID="1a294a5e-b315-4c9e-82f3-47f5dd2d729e" containerID="79a79e67b543fd4145aa34a8b4a4f38b6f96df118e832eedfacd940963da9b19" exitCode=60 Nov 03 00:00:04.375763 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.375673 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs" event={"ID":"1a294a5e-b315-4c9e-82f3-47f5dd2d729e","Type":"ContainerDied","Data":"79a79e67b543fd4145aa34a8b4a4f38b6f96df118e832eedfacd940963da9b19"} Nov 03 00:00:04.375945 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.375929 5114 scope.go:122] "RemoveContainer" containerID="79a79e67b543fd4145aa34a8b4a4f38b6f96df118e832eedfacd940963da9b19" Nov 03 00:00:04.376167 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:00:04.376148 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs_odh-ai-gateway-infra(1a294a5e-b315-4c9e-82f3-47f5dd2d729e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs" podUID="1a294a5e-b315-4c9e-82f3-47f5dd2d729e" Nov 03 00:00:04.377167 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.377154 5114 scope.go:122] "RemoveContainer" containerID="dd3505186bb738fcb1c0c83eb28d8723467e7308c4d5e7d0d58d5d3e9ded6a48" Nov 03 00:00:04.390338 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.390316 5114 scope.go:122] "RemoveContainer" containerID="0f2f578379e54c3e4c4d5c7f898b9b709f35e570da216319c1fb8e7cc3b358b1" Nov 03 00:00:04.404008 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.403978 5114 scope.go:122] "RemoveContainer" containerID="292ad08dd9d4545136efa1822b8225e9754f76a68ed045f095d87ced6245a573" Nov 03 00:00:04.473994 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.473971 5114 scope.go:122] "RemoveContainer" containerID="407e555b3046a999de96b9071f4792e15e6c2ccd47f7717488482459d17edb78" Nov 03 00:00:04.487697 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:04.487670 5114 scope.go:122] "RemoveContainer" containerID="efd07f6dd54e11410ce2dc4bf5a9e0d5636d085694eee58bbc346eb76b498fdd" Nov 03 00:00:05.383909 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:05.383878 5114 scope.go:122] "RemoveContainer" containerID="0501005e23821c03c16aa70b49981ba0c71f97b35de7cd78e1d613ec934d60f9" Nov 03 00:00:05.384410 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:00:05.384188 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf_odh-ai-gateway-infra(89ca75ec-2516-4f65-b600-4b6d9c72908d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf" podUID="89ca75ec-2516-4f65-b600-4b6d9c72908d" Nov 03 00:00:05.385646 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:05.385623 5114 scope.go:122] "RemoveContainer" containerID="c291c2fdce700f77b69f57400c1c61c3cd7c9b8a3832b0b1c02c898a528c8e7b" Nov 03 00:00:05.385945 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:00:05.385925 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw_odh-ai-gateway-infra(53b68115-0b68-4d74-96ec-e70f18391969)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw" podUID="53b68115-0b68-4d74-96ec-e70f18391969" Nov 03 00:00:05.387259 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:05.387238 5114 scope.go:122] "RemoveContainer" containerID="12151789d18a774582344f6578b73d0fb47d727b79fc3d750a721cce9819d67d" Nov 03 00:00:05.387540 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:00:05.387516 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx_odh-ai-gateway-infra(ccf010d7-6b97-4824-83da-9acac85d87bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx" podUID="ccf010d7-6b97-4824-83da-9acac85d87bd" Nov 03 00:00:05.388796 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:05.388773 5114 scope.go:122] "RemoveContainer" containerID="a0936bf0fde8fde82002cc02263f3a349ff2a97466633e4008d33678c7f57eb2" Nov 03 00:00:05.389070 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:00:05.389039 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n_odh-ai-gateway-infra(730f6d5e-c29d-491d-b40d-197ee83b76ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n" podUID="730f6d5e-c29d-491d-b40d-197ee83b76ff" Nov 03 00:00:05.390396 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:05.390373 5114 generic.go:356] "Generic (PLEG): container finished" podID="2da46d94-2113-49ed-8349-d0a0f38cde59" containerID="e212400dbe0062fbdb35a7f3475db30dbc7390386b3f9cfe6f787bcfb99bcdba" exitCode=60 Nov 03 00:00:05.390524 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:05.390457 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd" event={"ID":"2da46d94-2113-49ed-8349-d0a0f38cde59","Type":"ContainerDied","Data":"e212400dbe0062fbdb35a7f3475db30dbc7390386b3f9cfe6f787bcfb99bcdba"} Nov 03 00:00:05.390613 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:05.390574 5114 scope.go:122] "RemoveContainer" containerID="4128e57631f02a1ac2bacd9c6a084b3bef36eb29fe0e36f9510caeacd337703d" Nov 03 00:00:05.390696 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:05.390676 5114 scope.go:122] "RemoveContainer" containerID="e212400dbe0062fbdb35a7f3475db30dbc7390386b3f9cfe6f787bcfb99bcdba" Nov 03 00:00:05.390952 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:00:05.390927 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd_odh-ai-gateway-infra(2da46d94-2113-49ed-8349-d0a0f38cde59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd" podUID="2da46d94-2113-49ed-8349-d0a0f38cde59" Nov 03 00:00:05.392343 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:05.392327 5114 scope.go:122] "RemoveContainer" containerID="79a79e67b543fd4145aa34a8b4a4f38b6f96df118e832eedfacd940963da9b19" Nov 03 00:00:05.392612 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:00:05.392593 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs_odh-ai-gateway-infra(1a294a5e-b315-4c9e-82f3-47f5dd2d729e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs" podUID="1a294a5e-b315-4c9e-82f3-47f5dd2d729e" Nov 03 00:00:05.393854 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:05.393839 5114 scope.go:122] "RemoveContainer" containerID="b7a800731fd9ac2ebcb87dff06052079dc960bea279ce1c61ffca2f71027fbc6" Nov 03 00:00:05.394044 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:00:05.394029 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m_odh-ai-gateway-infra(5762c029-f662-41b5-9941-86af30749b7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m" podUID="5762c029-f662-41b5-9941-86af30749b7d" Nov 03 00:00:06.401194 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:06.401162 5114 scope.go:122] "RemoveContainer" containerID="e212400dbe0062fbdb35a7f3475db30dbc7390386b3f9cfe6f787bcfb99bcdba" Nov 03 00:00:06.401629 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:00:06.401446 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd_odh-ai-gateway-infra(2da46d94-2113-49ed-8349-d0a0f38cde59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd" podUID="2da46d94-2113-49ed-8349-d0a0f38cde59" Nov 03 00:00:07.584098 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:00:07.584057 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:00:14.812315 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:14.811827 5114 scope.go:122] "RemoveContainer" containerID="c291c2fdce700f77b69f57400c1c61c3cd7c9b8a3832b0b1c02c898a528c8e7b" Nov 03 00:00:14.828336 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:14.818103 5114 scope.go:122] "RemoveContainer" containerID="0501005e23821c03c16aa70b49981ba0c71f97b35de7cd78e1d613ec934d60f9" Nov 03 00:00:15.111633 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:15.111583 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" podUID="5e3275e5-f49f-4ca7-8d49-03178a77fee6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:00:15.260506 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:15.260122 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" podUID="bcd6362b-2995-4ebc-972d-dfcee6e41357" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:00:15.305530 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:15.301014 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" podUID="496c6815-1b56-4082-9ffd-9ea51a762a58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:00:15.362587 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:15.358726 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" podUID="fb16fc7d-3def-41dc-91a7-1103ccaf4c35" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:00:15.362587 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:15.361581 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" podUID="72301e09-ed03-4bc9-8264-028de7930a48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:00:15.587375 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:15.587335 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf" event={"ID":"89ca75ec-2516-4f65-b600-4b6d9c72908d","Type":"ContainerStarted","Data":"e3b7ea1b77cec7e349bd9546b80198cc0cc126bb2bf67132cb104e985af81929"} Nov 03 00:00:15.590202 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:15.590154 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw" event={"ID":"53b68115-0b68-4d74-96ec-e70f18391969","Type":"ContainerStarted","Data":"e12d550dd0838240672663f2b1d79e872fb8a1c1680d46d7eb1f3d1143273215"} Nov 03 00:00:15.787563 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:15.787531 5114 scope.go:122] "RemoveContainer" containerID="79a79e67b543fd4145aa34a8b4a4f38b6f96df118e832eedfacd940963da9b19" Nov 03 00:00:15.787979 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:15.787958 5114 scope.go:122] "RemoveContainer" containerID="12151789d18a774582344f6578b73d0fb47d727b79fc3d750a721cce9819d67d" Nov 03 00:00:15.788070 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:15.788026 5114 scope.go:122] "RemoveContainer" containerID="e212400dbe0062fbdb35a7f3475db30dbc7390386b3f9cfe6f787bcfb99bcdba" Nov 03 00:00:16.597965 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:16.597927 5114 generic.go:356] "Generic (PLEG): container finished" podID="89ca75ec-2516-4f65-b600-4b6d9c72908d" containerID="e3b7ea1b77cec7e349bd9546b80198cc0cc126bb2bf67132cb104e985af81929" exitCode=60 Nov 03 00:00:16.598508 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:16.597971 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf" event={"ID":"89ca75ec-2516-4f65-b600-4b6d9c72908d","Type":"ContainerDied","Data":"e3b7ea1b77cec7e349bd9546b80198cc0cc126bb2bf67132cb104e985af81929"} Nov 03 00:00:16.598508 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:16.598042 5114 scope.go:122] "RemoveContainer" containerID="0501005e23821c03c16aa70b49981ba0c71f97b35de7cd78e1d613ec934d60f9" Nov 03 00:00:16.598508 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:16.598437 5114 scope.go:122] "RemoveContainer" containerID="e3b7ea1b77cec7e349bd9546b80198cc0cc126bb2bf67132cb104e985af81929" Nov 03 00:00:16.598768 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:00:16.598749 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf_odh-ai-gateway-infra(89ca75ec-2516-4f65-b600-4b6d9c72908d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf" podUID="89ca75ec-2516-4f65-b600-4b6d9c72908d" Nov 03 00:00:16.601033 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:16.601011 5114 generic.go:356] "Generic (PLEG): container finished" podID="53b68115-0b68-4d74-96ec-e70f18391969" containerID="e12d550dd0838240672663f2b1d79e872fb8a1c1680d46d7eb1f3d1143273215" exitCode=60 Nov 03 00:00:16.601156 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:16.601073 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw" event={"ID":"53b68115-0b68-4d74-96ec-e70f18391969","Type":"ContainerDied","Data":"e12d550dd0838240672663f2b1d79e872fb8a1c1680d46d7eb1f3d1143273215"} Nov 03 00:00:16.601517 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:16.601470 5114 scope.go:122] "RemoveContainer" containerID="e12d550dd0838240672663f2b1d79e872fb8a1c1680d46d7eb1f3d1143273215" Nov 03 00:00:16.601830 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:00:16.601812 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw_odh-ai-gateway-infra(53b68115-0b68-4d74-96ec-e70f18391969)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw" podUID="53b68115-0b68-4d74-96ec-e70f18391969" Nov 03 00:00:16.603320 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:16.603296 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx" event={"ID":"ccf010d7-6b97-4824-83da-9acac85d87bd","Type":"ContainerStarted","Data":"c1ac9c7f03edde2520fafa223bd5cb1ee3b64f079837e76c835a1923a6d91c0c"} Nov 03 00:00:16.605421 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:16.605397 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd" event={"ID":"2da46d94-2113-49ed-8349-d0a0f38cde59","Type":"ContainerStarted","Data":"b6b04555c0db53ae29cdf7f7b0e3034921a88ca10bad58f0fb17604a6e1600ac"} Nov 03 00:00:16.610530 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:16.608653 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs" event={"ID":"1a294a5e-b315-4c9e-82f3-47f5dd2d729e","Type":"ContainerStarted","Data":"05629b158c6d970c9485a380f6845a3f0a5714c5ef1d6ed9cd5c0a72a3b61d06"} Nov 03 00:00:16.613966 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:16.613943 5114 scope.go:122] "RemoveContainer" containerID="c291c2fdce700f77b69f57400c1c61c3cd7c9b8a3832b0b1c02c898a528c8e7b" Nov 03 00:00:16.713184 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:16.713148 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf"] Nov 03 00:00:16.787622 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:16.787590 5114 scope.go:122] "RemoveContainer" containerID="a0936bf0fde8fde82002cc02263f3a349ff2a97466633e4008d33678c7f57eb2" Nov 03 00:00:16.827600 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:16.827553 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw"] Nov 03 00:00:17.620243 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.620214 5114 generic.go:356] "Generic (PLEG): container finished" podID="ccf010d7-6b97-4824-83da-9acac85d87bd" containerID="c1ac9c7f03edde2520fafa223bd5cb1ee3b64f079837e76c835a1923a6d91c0c" exitCode=60 Nov 03 00:00:17.620669 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.620261 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx" event={"ID":"ccf010d7-6b97-4824-83da-9acac85d87bd","Type":"ContainerDied","Data":"c1ac9c7f03edde2520fafa223bd5cb1ee3b64f079837e76c835a1923a6d91c0c"} Nov 03 00:00:17.620669 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.620346 5114 scope.go:122] "RemoveContainer" containerID="12151789d18a774582344f6578b73d0fb47d727b79fc3d750a721cce9819d67d" Nov 03 00:00:17.620927 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.620906 5114 scope.go:122] "RemoveContainer" containerID="c1ac9c7f03edde2520fafa223bd5cb1ee3b64f079837e76c835a1923a6d91c0c" Nov 03 00:00:17.621300 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:00:17.621268 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx_odh-ai-gateway-infra(ccf010d7-6b97-4824-83da-9acac85d87bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx" podUID="ccf010d7-6b97-4824-83da-9acac85d87bd" Nov 03 00:00:17.622151 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.622123 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n" event={"ID":"730f6d5e-c29d-491d-b40d-197ee83b76ff","Type":"ContainerStarted","Data":"2827435a51d2180b1963c85fef16d23e5b7221b43f7c030a78fe07cc54717c9c"} Nov 03 00:00:17.624641 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.624615 5114 generic.go:356] "Generic (PLEG): container finished" podID="2da46d94-2113-49ed-8349-d0a0f38cde59" containerID="b6b04555c0db53ae29cdf7f7b0e3034921a88ca10bad58f0fb17604a6e1600ac" exitCode=60 Nov 03 00:00:17.624742 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.624661 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd" event={"ID":"2da46d94-2113-49ed-8349-d0a0f38cde59","Type":"ContainerDied","Data":"b6b04555c0db53ae29cdf7f7b0e3034921a88ca10bad58f0fb17604a6e1600ac"} Nov 03 00:00:17.625063 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.625040 5114 scope.go:122] "RemoveContainer" containerID="b6b04555c0db53ae29cdf7f7b0e3034921a88ca10bad58f0fb17604a6e1600ac" Nov 03 00:00:17.625405 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:00:17.625376 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd_odh-ai-gateway-infra(2da46d94-2113-49ed-8349-d0a0f38cde59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd" podUID="2da46d94-2113-49ed-8349-d0a0f38cde59" Nov 03 00:00:17.628941 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.628914 5114 generic.go:356] "Generic (PLEG): container finished" podID="1a294a5e-b315-4c9e-82f3-47f5dd2d729e" containerID="05629b158c6d970c9485a380f6845a3f0a5714c5ef1d6ed9cd5c0a72a3b61d06" exitCode=60 Nov 03 00:00:17.629070 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.628988 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs" event={"ID":"1a294a5e-b315-4c9e-82f3-47f5dd2d729e","Type":"ContainerDied","Data":"05629b158c6d970c9485a380f6845a3f0a5714c5ef1d6ed9cd5c0a72a3b61d06"} Nov 03 00:00:17.629374 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.629358 5114 scope.go:122] "RemoveContainer" containerID="05629b158c6d970c9485a380f6845a3f0a5714c5ef1d6ed9cd5c0a72a3b61d06" Nov 03 00:00:17.629701 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:00:17.629672 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs_odh-ai-gateway-infra(1a294a5e-b315-4c9e-82f3-47f5dd2d729e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs" podUID="1a294a5e-b315-4c9e-82f3-47f5dd2d729e" Nov 03 00:00:17.635363 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.635342 5114 scope.go:122] "RemoveContainer" containerID="e212400dbe0062fbdb35a7f3475db30dbc7390386b3f9cfe6f787bcfb99bcdba" Nov 03 00:00:17.650663 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.650638 5114 scope.go:122] "RemoveContainer" containerID="79a79e67b543fd4145aa34a8b4a4f38b6f96df118e832eedfacd940963da9b19" Nov 03 00:00:17.787871 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.787380 5114 scope.go:122] "RemoveContainer" containerID="b7a800731fd9ac2ebcb87dff06052079dc960bea279ce1c61ffca2f71027fbc6" Nov 03 00:00:17.848494 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.848443 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx"] Nov 03 00:00:17.853586 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.853553 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf" Nov 03 00:00:17.856118 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.856099 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw" Nov 03 00:00:17.907346 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.907256 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53b68115-0b68-4d74-96ec-e70f18391969-kube-api-access-hnq5r\" (UniqueName: \"kubernetes.io/projected/53b68115-0b68-4d74-96ec-e70f18391969-kube-api-access-hnq5r\") pod \"53b68115-0b68-4d74-96ec-e70f18391969\" (UID: \"53b68115-0b68-4d74-96ec-e70f18391969\") " Nov 03 00:00:17.907346 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.907307 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89ca75ec-2516-4f65-b600-4b6d9c72908d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89ca75ec-2516-4f65-b600-4b6d9c72908d-maas-api-service-ca\") pod \"89ca75ec-2516-4f65-b600-4b6d9c72908d\" (UID: \"89ca75ec-2516-4f65-b600-4b6d9c72908d\") " Nov 03 00:00:17.907569 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.907410 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89ca75ec-2516-4f65-b600-4b6d9c72908d-kube-api-access-4kk5q\" (UniqueName: \"kubernetes.io/projected/89ca75ec-2516-4f65-b600-4b6d9c72908d-kube-api-access-4kk5q\") pod \"89ca75ec-2516-4f65-b600-4b6d9c72908d\" (UID: \"89ca75ec-2516-4f65-b600-4b6d9c72908d\") " Nov 03 00:00:17.907569 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.907475 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53b68115-0b68-4d74-96ec-e70f18391969-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b68115-0b68-4d74-96ec-e70f18391969-maas-api-service-ca\") pod \"53b68115-0b68-4d74-96ec-e70f18391969\" (UID: \"53b68115-0b68-4d74-96ec-e70f18391969\") " Nov 03 00:00:17.907827 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.907790 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89ca75ec-2516-4f65-b600-4b6d9c72908d-maas-api-service-ca" pod "89ca75ec-2516-4f65-b600-4b6d9c72908d" (UID: "89ca75ec-2516-4f65-b600-4b6d9c72908d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:17.907965 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.907819 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53b68115-0b68-4d74-96ec-e70f18391969-maas-api-service-ca" pod "53b68115-0b68-4d74-96ec-e70f18391969" (UID: "53b68115-0b68-4d74-96ec-e70f18391969"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:17.908110 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.908094 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b68115-0b68-4d74-96ec-e70f18391969-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:17.908166 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.908118 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89ca75ec-2516-4f65-b600-4b6d9c72908d-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:17.909845 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.909796 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ca75ec-2516-4f65-b600-4b6d9c72908d-kube-api-access-4kk5q" pod "89ca75ec-2516-4f65-b600-4b6d9c72908d" (UID: "89ca75ec-2516-4f65-b600-4b6d9c72908d"). InnerVolumeSpecName "kube-api-access-4kk5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:17.910108 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.910087 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b68115-0b68-4d74-96ec-e70f18391969-kube-api-access-hnq5r" pod "53b68115-0b68-4d74-96ec-e70f18391969" (UID: "53b68115-0b68-4d74-96ec-e70f18391969"). InnerVolumeSpecName "kube-api-access-hnq5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:17.961224 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:17.961184 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd"] Nov 03 00:00:18.008994 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.008950 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kk5q\" (UniqueName: \"kubernetes.io/projected/89ca75ec-2516-4f65-b600-4b6d9c72908d-kube-api-access-4kk5q\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:18.008994 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.008994 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnq5r\" (UniqueName: \"kubernetes.io/projected/53b68115-0b68-4d74-96ec-e70f18391969-kube-api-access-hnq5r\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:18.188411 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.188318 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs"] Nov 03 00:00:18.639465 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.639436 5114 generic.go:356] "Generic (PLEG): container finished" podID="5762c029-f662-41b5-9941-86af30749b7d" containerID="12984decb0bc8aca548527a9bd35a91810cba38a4537223af34b8760f25eb140" exitCode=60 Nov 03 00:00:18.639911 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.639525 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m" event={"ID":"5762c029-f662-41b5-9941-86af30749b7d","Type":"ContainerDied","Data":"12984decb0bc8aca548527a9bd35a91810cba38a4537223af34b8760f25eb140"} Nov 03 00:00:18.639911 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.639595 5114 scope.go:122] "RemoveContainer" containerID="b7a800731fd9ac2ebcb87dff06052079dc960bea279ce1c61ffca2f71027fbc6" Nov 03 00:00:18.640036 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.639992 5114 scope.go:122] "RemoveContainer" containerID="12984decb0bc8aca548527a9bd35a91810cba38a4537223af34b8760f25eb140" Nov 03 00:00:18.640357 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:00:18.640329 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m_odh-ai-gateway-infra(5762c029-f662-41b5-9941-86af30749b7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m" podUID="5762c029-f662-41b5-9941-86af30749b7d" Nov 03 00:00:18.642813 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.642779 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf" event={"ID":"89ca75ec-2516-4f65-b600-4b6d9c72908d","Type":"ContainerDied","Data":"849a699d8b8d996acef6d7c479e6671ceaed4877caba35da3fa1f2f82a54ca03"} Nov 03 00:00:18.642944 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.642891 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf" Nov 03 00:00:18.646937 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.646906 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw" event={"ID":"53b68115-0b68-4d74-96ec-e70f18391969","Type":"ContainerDied","Data":"d8c764ae72181dff1cb6b0f6f28ebacf59fce1661eae9af20df792be790b4ac5"} Nov 03 00:00:18.647059 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.646914 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw" Nov 03 00:00:18.654718 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.654685 5114 generic.go:356] "Generic (PLEG): container finished" podID="730f6d5e-c29d-491d-b40d-197ee83b76ff" containerID="2827435a51d2180b1963c85fef16d23e5b7221b43f7c030a78fe07cc54717c9c" exitCode=60 Nov 03 00:00:18.654837 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.654807 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n" event={"ID":"730f6d5e-c29d-491d-b40d-197ee83b76ff","Type":"ContainerDied","Data":"2827435a51d2180b1963c85fef16d23e5b7221b43f7c030a78fe07cc54717c9c"} Nov 03 00:00:18.655232 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.655209 5114 scope.go:122] "RemoveContainer" containerID="2827435a51d2180b1963c85fef16d23e5b7221b43f7c030a78fe07cc54717c9c" Nov 03 00:00:18.655554 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:00:18.655531 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n_odh-ai-gateway-infra(730f6d5e-c29d-491d-b40d-197ee83b76ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n" podUID="730f6d5e-c29d-491d-b40d-197ee83b76ff" Nov 03 00:00:18.664110 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.664082 5114 scope.go:122] "RemoveContainer" containerID="e3b7ea1b77cec7e349bd9546b80198cc0cc126bb2bf67132cb104e985af81929" Nov 03 00:00:18.676831 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.676801 5114 scope.go:122] "RemoveContainer" containerID="e12d550dd0838240672663f2b1d79e872fb8a1c1680d46d7eb1f3d1143273215" Nov 03 00:00:18.692515 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.692379 5114 scope.go:122] "RemoveContainer" containerID="a0936bf0fde8fde82002cc02263f3a349ff2a97466633e4008d33678c7f57eb2" Nov 03 00:00:18.825149 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.825084 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs" Nov 03 00:00:18.893713 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.893682 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx" Nov 03 00:00:18.897870 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.897845 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd" Nov 03 00:00:18.919117 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.919079 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a294a5e-b315-4c9e-82f3-47f5dd2d729e-kube-api-access-28cx2\" (UniqueName: \"kubernetes.io/projected/1a294a5e-b315-4c9e-82f3-47f5dd2d729e-kube-api-access-28cx2\") pod \"1a294a5e-b315-4c9e-82f3-47f5dd2d729e\" (UID: \"1a294a5e-b315-4c9e-82f3-47f5dd2d729e\") " Nov 03 00:00:18.919297 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.919167 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf010d7-6b97-4824-83da-9acac85d87bd-kube-api-access-cfxjz\" (UniqueName: \"kubernetes.io/projected/ccf010d7-6b97-4824-83da-9acac85d87bd-kube-api-access-cfxjz\") pod \"ccf010d7-6b97-4824-83da-9acac85d87bd\" (UID: \"ccf010d7-6b97-4824-83da-9acac85d87bd\") " Nov 03 00:00:18.919297 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.919215 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccf010d7-6b97-4824-83da-9acac85d87bd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccf010d7-6b97-4824-83da-9acac85d87bd-maas-api-service-ca\") pod \"ccf010d7-6b97-4824-83da-9acac85d87bd\" (UID: \"ccf010d7-6b97-4824-83da-9acac85d87bd\") " Nov 03 00:00:18.919297 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.919235 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a294a5e-b315-4c9e-82f3-47f5dd2d729e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a294a5e-b315-4c9e-82f3-47f5dd2d729e-maas-api-service-ca\") pod \"1a294a5e-b315-4c9e-82f3-47f5dd2d729e\" (UID: \"1a294a5e-b315-4c9e-82f3-47f5dd2d729e\") " Nov 03 00:00:18.919610 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.919579 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccf010d7-6b97-4824-83da-9acac85d87bd-maas-api-service-ca" pod "ccf010d7-6b97-4824-83da-9acac85d87bd" (UID: "ccf010d7-6b97-4824-83da-9acac85d87bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:18.919724 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.919670 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a294a5e-b315-4c9e-82f3-47f5dd2d729e-maas-api-service-ca" pod "1a294a5e-b315-4c9e-82f3-47f5dd2d729e" (UID: "1a294a5e-b315-4c9e-82f3-47f5dd2d729e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:18.919774 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.919754 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccf010d7-6b97-4824-83da-9acac85d87bd-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:18.919809 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.919778 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a294a5e-b315-4c9e-82f3-47f5dd2d729e-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:18.921558 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.921541 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf010d7-6b97-4824-83da-9acac85d87bd-kube-api-access-cfxjz" pod "ccf010d7-6b97-4824-83da-9acac85d87bd" (UID: "ccf010d7-6b97-4824-83da-9acac85d87bd"). InnerVolumeSpecName "kube-api-access-cfxjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:18.921627 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.921604 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a294a5e-b315-4c9e-82f3-47f5dd2d729e-kube-api-access-28cx2" pod "1a294a5e-b315-4c9e-82f3-47f5dd2d729e" (UID: "1a294a5e-b315-4c9e-82f3-47f5dd2d729e"). InnerVolumeSpecName "kube-api-access-28cx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:18.974759 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:18.974719 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n"] Nov 03 00:00:19.020827 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.020794 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da46d94-2113-49ed-8349-d0a0f38cde59-kube-api-access-mhrjb\" (UniqueName: \"kubernetes.io/projected/2da46d94-2113-49ed-8349-d0a0f38cde59-kube-api-access-mhrjb\") pod \"2da46d94-2113-49ed-8349-d0a0f38cde59\" (UID: \"2da46d94-2113-49ed-8349-d0a0f38cde59\") " Nov 03 00:00:19.021014 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.020872 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2da46d94-2113-49ed-8349-d0a0f38cde59-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da46d94-2113-49ed-8349-d0a0f38cde59-maas-api-service-ca\") pod \"2da46d94-2113-49ed-8349-d0a0f38cde59\" (UID: \"2da46d94-2113-49ed-8349-d0a0f38cde59\") " Nov 03 00:00:19.021321 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.021288 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da46d94-2113-49ed-8349-d0a0f38cde59-maas-api-service-ca" pod "2da46d94-2113-49ed-8349-d0a0f38cde59" (UID: "2da46d94-2113-49ed-8349-d0a0f38cde59"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:19.021462 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.021336 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28cx2\" (UniqueName: \"kubernetes.io/projected/1a294a5e-b315-4c9e-82f3-47f5dd2d729e-kube-api-access-28cx2\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:19.021462 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.021360 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfxjz\" (UniqueName: \"kubernetes.io/projected/ccf010d7-6b97-4824-83da-9acac85d87bd-kube-api-access-cfxjz\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:19.023339 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.023310 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da46d94-2113-49ed-8349-d0a0f38cde59-kube-api-access-mhrjb" pod "2da46d94-2113-49ed-8349-d0a0f38cde59" (UID: "2da46d94-2113-49ed-8349-d0a0f38cde59"). InnerVolumeSpecName "kube-api-access-mhrjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:19.099400 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.099367 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf"] Nov 03 00:00:19.108742 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.108706 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-440c60-29894400-6c2tf"] Nov 03 00:00:19.123026 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.122939 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhrjb\" (UniqueName: \"kubernetes.io/projected/2da46d94-2113-49ed-8349-d0a0f38cde59-kube-api-access-mhrjb\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:19.123026 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.122972 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da46d94-2113-49ed-8349-d0a0f38cde59-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:19.217300 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.217264 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw"] Nov 03 00:00:19.221556 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.221521 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5db41e-29894400-xlzgw"] Nov 03 00:00:19.669748 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.669701 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx" event={"ID":"ccf010d7-6b97-4824-83da-9acac85d87bd","Type":"ContainerDied","Data":"517e9050dce4d4783499299adf1ca9d2a4c21676c2eeb1c0146b64f4b2453132"} Nov 03 00:00:19.670267 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.669716 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx" Nov 03 00:00:19.670267 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.669806 5114 scope.go:122] "RemoveContainer" containerID="c1ac9c7f03edde2520fafa223bd5cb1ee3b64f079837e76c835a1923a6d91c0c" Nov 03 00:00:19.674515 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.674451 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd" Nov 03 00:00:19.674515 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.674444 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd" event={"ID":"2da46d94-2113-49ed-8349-d0a0f38cde59","Type":"ContainerDied","Data":"980d1a55367161e586686d65227c59d3bb9d71005e39d9c6e22702a19b95b6de"} Nov 03 00:00:19.676350 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.676315 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs" event={"ID":"1a294a5e-b315-4c9e-82f3-47f5dd2d729e","Type":"ContainerDied","Data":"66dd1b38ed9c83d7f1cbae8ee84e83e507e00d5e38f5bd8726f0d07fcc38bb38"} Nov 03 00:00:19.676562 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.676464 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs" Nov 03 00:00:19.684596 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.684569 5114 scope.go:122] "RemoveContainer" containerID="b6b04555c0db53ae29cdf7f7b0e3034921a88ca10bad58f0fb17604a6e1600ac" Nov 03 00:00:19.699726 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.698589 5114 scope.go:122] "RemoveContainer" containerID="05629b158c6d970c9485a380f6845a3f0a5714c5ef1d6ed9cd5c0a72a3b61d06" Nov 03 00:00:19.848277 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.848251 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n" Nov 03 00:00:19.907800 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.907762 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs"] Nov 03 00:00:19.911580 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.911550 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c3d8e-29894400-42wbs"] Nov 03 00:00:19.934318 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.934229 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/730f6d5e-c29d-491d-b40d-197ee83b76ff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/730f6d5e-c29d-491d-b40d-197ee83b76ff-maas-api-service-ca\") pod \"730f6d5e-c29d-491d-b40d-197ee83b76ff\" (UID: \"730f6d5e-c29d-491d-b40d-197ee83b76ff\") " Nov 03 00:00:19.934318 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.934286 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/730f6d5e-c29d-491d-b40d-197ee83b76ff-kube-api-access-rbf6b\" (UniqueName: \"kubernetes.io/projected/730f6d5e-c29d-491d-b40d-197ee83b76ff-kube-api-access-rbf6b\") pod \"730f6d5e-c29d-491d-b40d-197ee83b76ff\" (UID: \"730f6d5e-c29d-491d-b40d-197ee83b76ff\") " Nov 03 00:00:19.934754 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.934726 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730f6d5e-c29d-491d-b40d-197ee83b76ff-maas-api-service-ca" pod "730f6d5e-c29d-491d-b40d-197ee83b76ff" (UID: "730f6d5e-c29d-491d-b40d-197ee83b76ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:19.936799 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:19.936771 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730f6d5e-c29d-491d-b40d-197ee83b76ff-kube-api-access-rbf6b" pod "730f6d5e-c29d-491d-b40d-197ee83b76ff" (UID: "730f6d5e-c29d-491d-b40d-197ee83b76ff"). InnerVolumeSpecName "kube-api-access-rbf6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:20.022462 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.022425 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx"] Nov 03 00:00:20.031013 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.030978 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894400-nwpdx"] Nov 03 00:00:20.036077 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.036050 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/730f6d5e-c29d-491d-b40d-197ee83b76ff-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:20.036077 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.036077 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbf6b\" (UniqueName: \"kubernetes.io/projected/730f6d5e-c29d-491d-b40d-197ee83b76ff-kube-api-access-rbf6b\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:20.141771 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.141731 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd"] Nov 03 00:00:20.146665 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.146636 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-08c2c6-29894400-8xbgd"] Nov 03 00:00:20.342929 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.342892 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m"] Nov 03 00:00:20.510849 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.510818 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m" Nov 03 00:00:20.542871 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.542833 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5762c029-f662-41b5-9941-86af30749b7d-kube-api-access-hg44s\" (UniqueName: \"kubernetes.io/projected/5762c029-f662-41b5-9941-86af30749b7d-kube-api-access-hg44s\") pod \"5762c029-f662-41b5-9941-86af30749b7d\" (UID: \"5762c029-f662-41b5-9941-86af30749b7d\") " Nov 03 00:00:20.543070 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.543055 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5762c029-f662-41b5-9941-86af30749b7d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5762c029-f662-41b5-9941-86af30749b7d-maas-api-service-ca\") pod \"5762c029-f662-41b5-9941-86af30749b7d\" (UID: \"5762c029-f662-41b5-9941-86af30749b7d\") " Nov 03 00:00:20.543462 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.543439 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5762c029-f662-41b5-9941-86af30749b7d-maas-api-service-ca" pod "5762c029-f662-41b5-9941-86af30749b7d" (UID: "5762c029-f662-41b5-9941-86af30749b7d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:20.545266 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.545243 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5762c029-f662-41b5-9941-86af30749b7d-kube-api-access-hg44s" pod "5762c029-f662-41b5-9941-86af30749b7d" (UID: "5762c029-f662-41b5-9941-86af30749b7d"). InnerVolumeSpecName "kube-api-access-hg44s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:20.645552 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.645452 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5762c029-f662-41b5-9941-86af30749b7d-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:20.645552 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.645508 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg44s\" (UniqueName: \"kubernetes.io/projected/5762c029-f662-41b5-9941-86af30749b7d-kube-api-access-hg44s\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:20.687221 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.687183 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m" Nov 03 00:00:20.687221 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.687198 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m" event={"ID":"5762c029-f662-41b5-9941-86af30749b7d","Type":"ContainerDied","Data":"5709d8595a8718474557b9ed0bc68fdb110bba58b3fdc97c3836a2b9d5440a74"} Nov 03 00:00:20.687793 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.687269 5114 scope.go:122] "RemoveContainer" containerID="12984decb0bc8aca548527a9bd35a91810cba38a4537223af34b8760f25eb140" Nov 03 00:00:20.690213 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.690185 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n" event={"ID":"730f6d5e-c29d-491d-b40d-197ee83b76ff","Type":"ContainerDied","Data":"0ba66d28e061ef8541c9a11d124246b09c12abe80efbf775c66ede49ea18fd76"} Nov 03 00:00:20.690350 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.690220 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n" Nov 03 00:00:20.698574 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.698545 5114 scope.go:122] "RemoveContainer" containerID="2827435a51d2180b1963c85fef16d23e5b7221b43f7c030a78fe07cc54717c9c" Nov 03 00:00:20.802031 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.797465 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a294a5e-b315-4c9e-82f3-47f5dd2d729e" path="/var/lib/kubelet/pods/1a294a5e-b315-4c9e-82f3-47f5dd2d729e/volumes" Nov 03 00:00:20.802031 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.798154 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2da46d94-2113-49ed-8349-d0a0f38cde59" path="/var/lib/kubelet/pods/2da46d94-2113-49ed-8349-d0a0f38cde59/volumes" Nov 03 00:00:20.802031 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.798802 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53b68115-0b68-4d74-96ec-e70f18391969" path="/var/lib/kubelet/pods/53b68115-0b68-4d74-96ec-e70f18391969/volumes" Nov 03 00:00:20.802031 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.799991 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89ca75ec-2516-4f65-b600-4b6d9c72908d" path="/var/lib/kubelet/pods/89ca75ec-2516-4f65-b600-4b6d9c72908d/volumes" Nov 03 00:00:20.802031 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.800775 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccf010d7-6b97-4824-83da-9acac85d87bd" path="/var/lib/kubelet/pods/ccf010d7-6b97-4824-83da-9acac85d87bd/volumes" Nov 03 00:00:20.850567 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.850527 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n"] Nov 03 00:00:20.855844 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.855807 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-882a2a-29894400-kjp7n"] Nov 03 00:00:20.964421 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.964327 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m"] Nov 03 00:00:20.968494 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:20.968448 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894400-gll8m"] Nov 03 00:00:22.793147 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:22.793113 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5762c029-f662-41b5-9941-86af30749b7d" path="/var/lib/kubelet/pods/5762c029-f662-41b5-9941-86af30749b7d/volumes" Nov 03 00:00:22.793570 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:22.793561 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="730f6d5e-c29d-491d-b40d-197ee83b76ff" path="/var/lib/kubelet/pods/730f6d5e-c29d-491d-b40d-197ee83b76ff/volumes" Nov 03 00:00:37.584983 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:00:37.584934 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:00:46.591892 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:46.591858 5114 scope.go:122] "RemoveContainer" containerID="553b9bd4f0b2120a0ebb836c1edfecdeac0cad05475b9e348d57c3c2e246a1a5" Nov 03 00:00:48.828098 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:48.828057 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254"] Nov 03 00:00:48.828606 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:48.828320 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" podUID="ce392789-cafb-4d1a-ac0e-05578d20785d" containerName="maas-api" containerID="cri-o://8326a6f74ca049756bd7060007c1d56d34aa3f1c856b82dc646a33a51a634bb9" gracePeriod=30 Nov 03 00:00:49.134205 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:49.134169 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" Nov 03 00:00:49.191455 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:49.191403 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce392789-cafb-4d1a-ac0e-05578d20785d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce392789-cafb-4d1a-ac0e-05578d20785d-maas-api-tls\") pod \"ce392789-cafb-4d1a-ac0e-05578d20785d\" (UID: \"ce392789-cafb-4d1a-ac0e-05578d20785d\") " Nov 03 00:00:49.191701 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:49.191525 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce392789-cafb-4d1a-ac0e-05578d20785d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce392789-cafb-4d1a-ac0e-05578d20785d-metrics-certs\") pod \"ce392789-cafb-4d1a-ac0e-05578d20785d\" (UID: \"ce392789-cafb-4d1a-ac0e-05578d20785d\") " Nov 03 00:00:49.191701 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:49.191597 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce392789-cafb-4d1a-ac0e-05578d20785d-kube-api-access-b9r4l\" (UniqueName: \"kubernetes.io/projected/ce392789-cafb-4d1a-ac0e-05578d20785d-kube-api-access-b9r4l\") pod \"ce392789-cafb-4d1a-ac0e-05578d20785d\" (UID: \"ce392789-cafb-4d1a-ac0e-05578d20785d\") " Nov 03 00:00:49.191701 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:49.191626 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce392789-cafb-4d1a-ac0e-05578d20785d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce392789-cafb-4d1a-ac0e-05578d20785d-openshift-service-ca\") pod \"ce392789-cafb-4d1a-ac0e-05578d20785d\" (UID: \"ce392789-cafb-4d1a-ac0e-05578d20785d\") " Nov 03 00:00:49.192144 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:49.192113 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce392789-cafb-4d1a-ac0e-05578d20785d-openshift-service-ca" pod "ce392789-cafb-4d1a-ac0e-05578d20785d" (UID: "ce392789-cafb-4d1a-ac0e-05578d20785d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:49.194528 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:49.194466 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce392789-cafb-4d1a-ac0e-05578d20785d-maas-api-tls" pod "ce392789-cafb-4d1a-ac0e-05578d20785d" (UID: "ce392789-cafb-4d1a-ac0e-05578d20785d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:00:49.194528 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:49.194519 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce392789-cafb-4d1a-ac0e-05578d20785d-kube-api-access-b9r4l" pod "ce392789-cafb-4d1a-ac0e-05578d20785d" (UID: "ce392789-cafb-4d1a-ac0e-05578d20785d"). InnerVolumeSpecName "kube-api-access-b9r4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:49.194701 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:49.194535 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce392789-cafb-4d1a-ac0e-05578d20785d-metrics-certs" pod "ce392789-cafb-4d1a-ac0e-05578d20785d" (UID: "ce392789-cafb-4d1a-ac0e-05578d20785d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:00:49.292775 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:49.292731 5114 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce392789-cafb-4d1a-ac0e-05578d20785d-metrics-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:49.292775 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:49.292773 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9r4l\" (UniqueName: \"kubernetes.io/projected/ce392789-cafb-4d1a-ac0e-05578d20785d-kube-api-access-b9r4l\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:49.292775 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:49.292789 5114 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce392789-cafb-4d1a-ac0e-05578d20785d-openshift-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:49.293067 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:49.292798 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce392789-cafb-4d1a-ac0e-05578d20785d-maas-api-tls\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:49.867192 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:49.867151 5114 generic.go:356] "Generic (PLEG): container finished" podID="ce392789-cafb-4d1a-ac0e-05578d20785d" containerID="8326a6f74ca049756bd7060007c1d56d34aa3f1c856b82dc646a33a51a634bb9" exitCode=0 Nov 03 00:00:49.867734 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:49.867209 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" Nov 03 00:00:49.867734 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:49.867233 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" event={"ID":"ce392789-cafb-4d1a-ac0e-05578d20785d","Type":"ContainerDied","Data":"8326a6f74ca049756bd7060007c1d56d34aa3f1c856b82dc646a33a51a634bb9"} Nov 03 00:00:49.867734 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:49.867300 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254" event={"ID":"ce392789-cafb-4d1a-ac0e-05578d20785d","Type":"ContainerDied","Data":"a5a2749c7fc70870e0c3524f29727e0139910c36b096baeea104f0c7843e7535"} Nov 03 00:00:49.867734 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:49.867325 5114 scope.go:122] "RemoveContainer" containerID="8326a6f74ca049756bd7060007c1d56d34aa3f1c856b82dc646a33a51a634bb9" Nov 03 00:00:49.885048 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:49.884861 5114 scope.go:122] "RemoveContainer" containerID="8326a6f74ca049756bd7060007c1d56d34aa3f1c856b82dc646a33a51a634bb9" Nov 03 00:00:49.885690 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:00:49.885661 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8326a6f74ca049756bd7060007c1d56d34aa3f1c856b82dc646a33a51a634bb9\": container with ID starting with 8326a6f74ca049756bd7060007c1d56d34aa3f1c856b82dc646a33a51a634bb9 not found: ID does not exist" containerID="8326a6f74ca049756bd7060007c1d56d34aa3f1c856b82dc646a33a51a634bb9" Nov 03 00:00:49.885781 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:49.885704 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8326a6f74ca049756bd7060007c1d56d34aa3f1c856b82dc646a33a51a634bb9"} err="failed to get container status \"8326a6f74ca049756bd7060007c1d56d34aa3f1c856b82dc646a33a51a634bb9\": rpc error: code = NotFound desc = could not find container \"8326a6f74ca049756bd7060007c1d56d34aa3f1c856b82dc646a33a51a634bb9\": container with ID starting with 8326a6f74ca049756bd7060007c1d56d34aa3f1c856b82dc646a33a51a634bb9 not found: ID does not exist" Nov 03 00:00:50.030611 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:50.030567 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254"] Nov 03 00:00:50.080103 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:50.080053 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5db41e-84ff7c899c-wp254"] Nov 03 00:00:50.796407 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:50.796371 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce392789-cafb-4d1a-ac0e-05578d20785d" path="/var/lib/kubelet/pods/ce392789-cafb-4d1a-ac0e-05578d20785d/volumes" Nov 03 00:00:50.826534 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:50.826497 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57"] Nov 03 00:00:50.876304 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:50.876268 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5db41e-7mv57"] Nov 03 00:00:52.793720 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:52.793684 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a4d12d-1da0-494b-a957-85600b3842f4" path="/var/lib/kubelet/pods/84a4d12d-1da0-494b-a957-85600b3842f4/volumes" Nov 03 00:00:53.644669 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.644630 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ed38add-m6dtb"] Nov 03 00:00:53.645091 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645076 5114 cpu_manager.go:414] "containerMap: removing container" podUID="89ca75ec-2516-4f65-b600-4b6d9c72908d" containerName="cleanup" Nov 03 00:00:53.645091 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645091 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89ca75ec-2516-4f65-b600-4b6d9c72908d" containerName="cleanup" Nov 03 00:00:53.645199 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645101 5114 cpu_manager.go:414] "containerMap: removing container" podUID="2da46d94-2113-49ed-8349-d0a0f38cde59" containerName="cleanup" Nov 03 00:00:53.645199 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645106 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da46d94-2113-49ed-8349-d0a0f38cde59" containerName="cleanup" Nov 03 00:00:53.645199 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645114 5114 cpu_manager.go:414] "containerMap: removing container" podUID="1a294a5e-b315-4c9e-82f3-47f5dd2d729e" containerName="cleanup" Nov 03 00:00:53.645199 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645119 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a294a5e-b315-4c9e-82f3-47f5dd2d729e" containerName="cleanup" Nov 03 00:00:53.645199 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645129 5114 cpu_manager.go:414] "containerMap: removing container" podUID="2da46d94-2113-49ed-8349-d0a0f38cde59" containerName="cleanup" Nov 03 00:00:53.645199 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645133 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da46d94-2113-49ed-8349-d0a0f38cde59" containerName="cleanup" Nov 03 00:00:53.645199 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645147 5114 cpu_manager.go:414] "containerMap: removing container" podUID="5762c029-f662-41b5-9941-86af30749b7d" containerName="cleanup" Nov 03 00:00:53.645199 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645154 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5762c029-f662-41b5-9941-86af30749b7d" containerName="cleanup" Nov 03 00:00:53.645199 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645165 5114 cpu_manager.go:414] "containerMap: removing container" podUID="730f6d5e-c29d-491d-b40d-197ee83b76ff" containerName="cleanup" Nov 03 00:00:53.645199 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645172 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730f6d5e-c29d-491d-b40d-197ee83b76ff" containerName="cleanup" Nov 03 00:00:53.645199 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645183 5114 cpu_manager.go:414] "containerMap: removing container" podUID="5762c029-f662-41b5-9941-86af30749b7d" containerName="cleanup" Nov 03 00:00:53.645199 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645190 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5762c029-f662-41b5-9941-86af30749b7d" containerName="cleanup" Nov 03 00:00:53.645199 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645201 5114 cpu_manager.go:414] "containerMap: removing container" podUID="ccf010d7-6b97-4824-83da-9acac85d87bd" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645208 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf010d7-6b97-4824-83da-9acac85d87bd" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645219 5114 cpu_manager.go:414] "containerMap: removing container" podUID="2da46d94-2113-49ed-8349-d0a0f38cde59" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645228 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da46d94-2113-49ed-8349-d0a0f38cde59" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645237 5114 cpu_manager.go:414] "containerMap: removing container" podUID="53b68115-0b68-4d74-96ec-e70f18391969" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645245 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b68115-0b68-4d74-96ec-e70f18391969" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645255 5114 cpu_manager.go:414] "containerMap: removing container" podUID="1a294a5e-b315-4c9e-82f3-47f5dd2d729e" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645261 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a294a5e-b315-4c9e-82f3-47f5dd2d729e" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645267 5114 cpu_manager.go:414] "containerMap: removing container" podUID="5762c029-f662-41b5-9941-86af30749b7d" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645273 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5762c029-f662-41b5-9941-86af30749b7d" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645280 5114 cpu_manager.go:414] "containerMap: removing container" podUID="89ca75ec-2516-4f65-b600-4b6d9c72908d" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645286 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89ca75ec-2516-4f65-b600-4b6d9c72908d" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645296 5114 cpu_manager.go:414] "containerMap: removing container" podUID="ce392789-cafb-4d1a-ac0e-05578d20785d" containerName="maas-api" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645301 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce392789-cafb-4d1a-ac0e-05578d20785d" containerName="maas-api" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645308 5114 cpu_manager.go:414] "containerMap: removing container" podUID="1a294a5e-b315-4c9e-82f3-47f5dd2d729e" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645312 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a294a5e-b315-4c9e-82f3-47f5dd2d729e" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645324 5114 cpu_manager.go:414] "containerMap: removing container" podUID="ccf010d7-6b97-4824-83da-9acac85d87bd" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645330 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf010d7-6b97-4824-83da-9acac85d87bd" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645344 5114 cpu_manager.go:414] "containerMap: removing container" podUID="730f6d5e-c29d-491d-b40d-197ee83b76ff" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645348 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730f6d5e-c29d-491d-b40d-197ee83b76ff" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645360 5114 cpu_manager.go:414] "containerMap: removing container" podUID="730f6d5e-c29d-491d-b40d-197ee83b76ff" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645364 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730f6d5e-c29d-491d-b40d-197ee83b76ff" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645376 5114 cpu_manager.go:414] "containerMap: removing container" podUID="53b68115-0b68-4d74-96ec-e70f18391969" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645380 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b68115-0b68-4d74-96ec-e70f18391969" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645456 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a294a5e-b315-4c9e-82f3-47f5dd2d729e" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645464 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da46d94-2113-49ed-8349-d0a0f38cde59" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645471 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="5762c029-f662-41b5-9941-86af30749b7d" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645494 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="730f6d5e-c29d-491d-b40d-197ee83b76ff" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645504 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="5762c029-f662-41b5-9941-86af30749b7d" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645512 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf010d7-6b97-4824-83da-9acac85d87bd" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645518 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da46d94-2113-49ed-8349-d0a0f38cde59" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645523 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b68115-0b68-4d74-96ec-e70f18391969" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645529 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="89ca75ec-2516-4f65-b600-4b6d9c72908d" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645535 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="5762c029-f662-41b5-9941-86af30749b7d" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645541 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="89ca75ec-2516-4f65-b600-4b6d9c72908d" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645547 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce392789-cafb-4d1a-ac0e-05578d20785d" containerName="maas-api" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645553 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a294a5e-b315-4c9e-82f3-47f5dd2d729e" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645559 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf010d7-6b97-4824-83da-9acac85d87bd" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645566 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf010d7-6b97-4824-83da-9acac85d87bd" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645572 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="730f6d5e-c29d-491d-b40d-197ee83b76ff" containerName="cleanup" Nov 03 00:00:53.645702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.645578 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b68115-0b68-4d74-96ec-e70f18391969" containerName="cleanup" Nov 03 00:00:53.648883 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.648853 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed38add-m6dtb" Nov 03 00:00:53.706329 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.706284 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ed38add-m6dtb"] Nov 03 00:00:53.751453 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.751394 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0e0537f-9d0d-4179-8329-85218b5c9949-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ed38add-m6dtb\" (UID: \"b0e0537f-9d0d-4179-8329-85218b5c9949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed38add-m6dtb" Nov 03 00:00:53.751674 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.751622 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2nmx\" (UniqueName: \"kubernetes.io/projected/b0e0537f-9d0d-4179-8329-85218b5c9949-kube-api-access-n2nmx\") pod \"maas-api-revoke-sub-1ed38add-m6dtb\" (UID: \"b0e0537f-9d0d-4179-8329-85218b5c9949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed38add-m6dtb" Nov 03 00:00:53.751674 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.751664 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0e0537f-9d0d-4179-8329-85218b5c9949-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ed38add-m6dtb\" (UID: \"b0e0537f-9d0d-4179-8329-85218b5c9949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed38add-m6dtb" Nov 03 00:00:53.752761 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.752740 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 03 00:00:53.752761 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.752753 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:00:53.853130 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.853087 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0e0537f-9d0d-4179-8329-85218b5c9949-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ed38add-m6dtb\" (UID: \"b0e0537f-9d0d-4179-8329-85218b5c9949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed38add-m6dtb" Nov 03 00:00:53.853627 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.853169 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2nmx\" (UniqueName: \"kubernetes.io/projected/b0e0537f-9d0d-4179-8329-85218b5c9949-kube-api-access-n2nmx\") pod \"maas-api-revoke-sub-1ed38add-m6dtb\" (UID: \"b0e0537f-9d0d-4179-8329-85218b5c9949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed38add-m6dtb" Nov 03 00:00:53.853627 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.853192 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0e0537f-9d0d-4179-8329-85218b5c9949-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ed38add-m6dtb\" (UID: \"b0e0537f-9d0d-4179-8329-85218b5c9949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed38add-m6dtb" Nov 03 00:00:53.853924 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.853903 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0e0537f-9d0d-4179-8329-85218b5c9949-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ed38add-m6dtb\" (UID: \"b0e0537f-9d0d-4179-8329-85218b5c9949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed38add-m6dtb" Nov 03 00:00:53.856649 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.856623 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0e0537f-9d0d-4179-8329-85218b5c9949-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ed38add-m6dtb\" (UID: \"b0e0537f-9d0d-4179-8329-85218b5c9949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed38add-m6dtb" Nov 03 00:00:53.914356 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.914265 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2nmx\" (UniqueName: \"kubernetes.io/projected/b0e0537f-9d0d-4179-8329-85218b5c9949-kube-api-access-n2nmx\") pod \"maas-api-revoke-sub-1ed38add-m6dtb\" (UID: \"b0e0537f-9d0d-4179-8329-85218b5c9949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed38add-m6dtb" Nov 03 00:00:53.962374 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:53.962333 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed38add-m6dtb" Nov 03 00:00:54.339608 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:54.339581 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ed38add-m6dtb"] Nov 03 00:00:54.341577 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:00:54.341535 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0e0537f_9d0d_4179_8329_85218b5c9949.slice/crio-b3287259443f437450e8576ed68bc150f97d98852b787fb6250cb2c503c5338a WatchSource:0}: Error finding container b3287259443f437450e8576ed68bc150f97d98852b787fb6250cb2c503c5338a: Status 404 returned error can't find the container with id b3287259443f437450e8576ed68bc150f97d98852b787fb6250cb2c503c5338a Nov 03 00:00:54.897468 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:54.897365 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed38add-m6dtb" event={"ID":"b0e0537f-9d0d-4179-8329-85218b5c9949","Type":"ContainerStarted","Data":"c9514fc2597ae010fd6d968db294103d7736e8352a8e68b05a7b8a06c897a3c8"} Nov 03 00:00:54.897468 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:54.897433 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed38add-m6dtb" event={"ID":"b0e0537f-9d0d-4179-8329-85218b5c9949","Type":"ContainerStarted","Data":"b3287259443f437450e8576ed68bc150f97d98852b787fb6250cb2c503c5338a"} Nov 03 00:00:55.012290 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:55.012160 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed38add-m6dtb" podStartSLOduration=2.012143332 podStartE2EDuration="2.012143332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:55.011958306 +0000 UTC m=+4330.364991685" watchObservedRunningTime="2026-11-03 00:00:55.012143332 +0000 UTC m=+4330.365176707" Nov 03 00:00:55.904757 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:55.904721 5114 generic.go:356] "Generic (PLEG): container finished" podID="b0e0537f-9d0d-4179-8329-85218b5c9949" containerID="c9514fc2597ae010fd6d968db294103d7736e8352a8e68b05a7b8a06c897a3c8" exitCode=0 Nov 03 00:00:55.905227 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:55.904795 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed38add-m6dtb" event={"ID":"b0e0537f-9d0d-4179-8329-85218b5c9949","Type":"ContainerDied","Data":"c9514fc2597ae010fd6d968db294103d7736e8352a8e68b05a7b8a06c897a3c8"} Nov 03 00:00:57.056136 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:57.056106 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed38add-m6dtb" Nov 03 00:00:57.193194 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:57.193090 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0e0537f-9d0d-4179-8329-85218b5c9949-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0e0537f-9d0d-4179-8329-85218b5c9949-maas-api-xks-ca\") pod \"b0e0537f-9d0d-4179-8329-85218b5c9949\" (UID: \"b0e0537f-9d0d-4179-8329-85218b5c9949\") " Nov 03 00:00:57.193385 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:57.193201 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0e0537f-9d0d-4179-8329-85218b5c9949-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0e0537f-9d0d-4179-8329-85218b5c9949-maas-api-openshift-ca\") pod \"b0e0537f-9d0d-4179-8329-85218b5c9949\" (UID: \"b0e0537f-9d0d-4179-8329-85218b5c9949\") " Nov 03 00:00:57.193385 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:57.193303 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0e0537f-9d0d-4179-8329-85218b5c9949-kube-api-access-n2nmx\" (UniqueName: \"kubernetes.io/projected/b0e0537f-9d0d-4179-8329-85218b5c9949-kube-api-access-n2nmx\") pod \"b0e0537f-9d0d-4179-8329-85218b5c9949\" (UID: \"b0e0537f-9d0d-4179-8329-85218b5c9949\") " Nov 03 00:00:57.193649 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:57.193623 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0e0537f-9d0d-4179-8329-85218b5c9949-maas-api-openshift-ca" pod "b0e0537f-9d0d-4179-8329-85218b5c9949" (UID: "b0e0537f-9d0d-4179-8329-85218b5c9949"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:57.193877 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:57.193858 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0e0537f-9d0d-4179-8329-85218b5c9949-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:57.195648 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:57.195619 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0e0537f-9d0d-4179-8329-85218b5c9949-kube-api-access-n2nmx" pod "b0e0537f-9d0d-4179-8329-85218b5c9949" (UID: "b0e0537f-9d0d-4179-8329-85218b5c9949"). InnerVolumeSpecName "kube-api-access-n2nmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:57.195808 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:57.195790 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0e0537f-9d0d-4179-8329-85218b5c9949-maas-api-xks-ca" pod "b0e0537f-9d0d-4179-8329-85218b5c9949" (UID: "b0e0537f-9d0d-4179-8329-85218b5c9949"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:00:57.295080 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:57.295036 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0e0537f-9d0d-4179-8329-85218b5c9949-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:57.295080 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:57.295075 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2nmx\" (UniqueName: \"kubernetes.io/projected/b0e0537f-9d0d-4179-8329-85218b5c9949-kube-api-access-n2nmx\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:57.917464 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:57.917425 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed38add-m6dtb" event={"ID":"b0e0537f-9d0d-4179-8329-85218b5c9949","Type":"ContainerDied","Data":"b3287259443f437450e8576ed68bc150f97d98852b787fb6250cb2c503c5338a"} Nov 03 00:00:57.917464 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:57.917461 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed38add-m6dtb" Nov 03 00:00:57.917743 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:00:57.917520 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3287259443f437450e8576ed68bc150f97d98852b787fb6250cb2c503c5338a" Nov 03 00:01:00.907608 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:00.907574 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w"] Nov 03 00:01:00.908091 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:00.907798 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" podUID="a53ad7b3-45b4-4450-a3c7-b7e00596d28d" containerName="istio-proxy" containerID="cri-o://93e5af1f99cbe6dcbb31a4fa2206feb2229626b5af523ef14a25cc8836723ae2" gracePeriod=30 Nov 03 00:01:04.577838 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:04.577792 5114 patch_prober.go:29] interesting pod/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:01:04.578354 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:04.577875 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" podUID="a53ad7b3-45b4-4450-a3c7-b7e00596d28d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:01:06.180812 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.180740 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 03 00:01:06.298833 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.298796 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-podinfo\") pod \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " Nov 03 00:01:06.298833 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.298846 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-data\") pod \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " Nov 03 00:01:06.299105 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.298902 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-token\" (UniqueName: \"kubernetes.io/projected/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-token\") pod \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " Nov 03 00:01:06.299105 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.298952 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-envoy\") pod \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " Nov 03 00:01:06.299105 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.299020 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-kube-api-access-c7zwq\" (UniqueName: \"kubernetes.io/projected/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-kube-api-access-c7zwq\") pod \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " Nov 03 00:01:06.299105 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.299079 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istiod-ca-cert\") pod \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " Nov 03 00:01:06.299337 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.299135 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-credential-socket\") pod \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " Nov 03 00:01:06.299337 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.299197 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-workload-certs\") pod \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " Nov 03 00:01:06.299337 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.299242 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-workload-socket\") pod \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\" (UID: \"a53ad7b3-45b4-4450-a3c7-b7e00596d28d\") " Nov 03 00:01:06.300326 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.299631 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-credential-socket" pod "a53ad7b3-45b4-4450-a3c7-b7e00596d28d" (UID: "a53ad7b3-45b4-4450-a3c7-b7e00596d28d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:01:06.300326 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.299788 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-data" pod "a53ad7b3-45b4-4450-a3c7-b7e00596d28d" (UID: "a53ad7b3-45b4-4450-a3c7-b7e00596d28d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:01:06.300326 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.299823 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:01:06.300326 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.299980 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-workload-socket" pod "a53ad7b3-45b4-4450-a3c7-b7e00596d28d" (UID: "a53ad7b3-45b4-4450-a3c7-b7e00596d28d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:01:06.300326 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.300256 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-workload-certs" pod "a53ad7b3-45b4-4450-a3c7-b7e00596d28d" (UID: "a53ad7b3-45b4-4450-a3c7-b7e00596d28d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:01:06.301549 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.301520 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istiod-ca-cert" pod "a53ad7b3-45b4-4450-a3c7-b7e00596d28d" (UID: "a53ad7b3-45b4-4450-a3c7-b7e00596d28d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:01:06.302471 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.302433 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-kube-api-access-c7zwq" pod "a53ad7b3-45b4-4450-a3c7-b7e00596d28d" (UID: "a53ad7b3-45b4-4450-a3c7-b7e00596d28d"). InnerVolumeSpecName "kube-api-access-c7zwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:01:06.302732 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.302707 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-envoy" pod "a53ad7b3-45b4-4450-a3c7-b7e00596d28d" (UID: "a53ad7b3-45b4-4450-a3c7-b7e00596d28d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:01:06.302994 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.302972 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-token" pod "a53ad7b3-45b4-4450-a3c7-b7e00596d28d" (UID: "a53ad7b3-45b4-4450-a3c7-b7e00596d28d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:01:06.303107 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.303086 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-podinfo" pod "a53ad7b3-45b4-4450-a3c7-b7e00596d28d" (UID: "a53ad7b3-45b4-4450-a3c7-b7e00596d28d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:01:06.401560 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.401519 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:01:06.401560 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.401555 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:01:06.401560 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.401565 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:01:06.401811 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.401575 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:01:06.401811 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.401585 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7zwq\" (UniqueName: \"kubernetes.io/projected/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-kube-api-access-c7zwq\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:01:06.401811 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.401594 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:01:06.401811 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.401603 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:01:06.401811 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.401613 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53ad7b3-45b4-4450-a3c7-b7e00596d28d-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:01:06.973281 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.973244 5114 generic.go:356] "Generic (PLEG): container finished" podID="a53ad7b3-45b4-4450-a3c7-b7e00596d28d" containerID="93e5af1f99cbe6dcbb31a4fa2206feb2229626b5af523ef14a25cc8836723ae2" exitCode=0 Nov 03 00:01:06.973466 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.973316 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" Nov 03 00:01:06.973466 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.973325 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" event={"ID":"a53ad7b3-45b4-4450-a3c7-b7e00596d28d","Type":"ContainerDied","Data":"93e5af1f99cbe6dcbb31a4fa2206feb2229626b5af523ef14a25cc8836723ae2"} Nov 03 00:01:06.973466 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.973373 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w" event={"ID":"a53ad7b3-45b4-4450-a3c7-b7e00596d28d","Type":"ContainerDied","Data":"d879703a5f01086b9352e081364c237309edab24bf8060a3de5f59e6aff8db4c"} Nov 03 00:01:06.973466 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.973408 5114 scope.go:122] "RemoveContainer" containerID="93e5af1f99cbe6dcbb31a4fa2206feb2229626b5af523ef14a25cc8836723ae2" Nov 03 00:01:06.983713 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.983692 5114 scope.go:122] "RemoveContainer" containerID="c8f3b0709d5781d8731b8bbed90346192d2c62de79b750a46ff9b1a1840f953e" Nov 03 00:01:06.993537 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.993515 5114 scope.go:122] "RemoveContainer" containerID="93e5af1f99cbe6dcbb31a4fa2206feb2229626b5af523ef14a25cc8836723ae2" Nov 03 00:01:06.993926 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:01:06.993897 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93e5af1f99cbe6dcbb31a4fa2206feb2229626b5af523ef14a25cc8836723ae2\": container with ID starting with 93e5af1f99cbe6dcbb31a4fa2206feb2229626b5af523ef14a25cc8836723ae2 not found: ID does not exist" containerID="93e5af1f99cbe6dcbb31a4fa2206feb2229626b5af523ef14a25cc8836723ae2" Nov 03 00:01:06.993981 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.993938 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93e5af1f99cbe6dcbb31a4fa2206feb2229626b5af523ef14a25cc8836723ae2"} err="failed to get container status \"93e5af1f99cbe6dcbb31a4fa2206feb2229626b5af523ef14a25cc8836723ae2\": rpc error: code = NotFound desc = could not find container \"93e5af1f99cbe6dcbb31a4fa2206feb2229626b5af523ef14a25cc8836723ae2\": container with ID starting with 93e5af1f99cbe6dcbb31a4fa2206feb2229626b5af523ef14a25cc8836723ae2 not found: ID does not exist" Nov 03 00:01:06.993981 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.993957 5114 scope.go:122] "RemoveContainer" containerID="c8f3b0709d5781d8731b8bbed90346192d2c62de79b750a46ff9b1a1840f953e" Nov 03 00:01:06.994252 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:01:06.994232 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f3b0709d5781d8731b8bbed90346192d2c62de79b750a46ff9b1a1840f953e\": container with ID starting with c8f3b0709d5781d8731b8bbed90346192d2c62de79b750a46ff9b1a1840f953e not found: ID does not exist" containerID="c8f3b0709d5781d8731b8bbed90346192d2c62de79b750a46ff9b1a1840f953e" Nov 03 00:01:06.994299 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:06.994258 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f3b0709d5781d8731b8bbed90346192d2c62de79b750a46ff9b1a1840f953e"} err="failed to get container status \"c8f3b0709d5781d8731b8bbed90346192d2c62de79b750a46ff9b1a1840f953e\": rpc error: code = NotFound desc = could not find container \"c8f3b0709d5781d8731b8bbed90346192d2c62de79b750a46ff9b1a1840f953e\": container with ID starting with c8f3b0709d5781d8731b8bbed90346192d2c62de79b750a46ff9b1a1840f953e not found: ID does not exist" Nov 03 00:01:07.138937 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:07.138842 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w"] Nov 03 00:01:07.145388 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:07.145356 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-5e7e9e-openshift-default-6c4f6d48cc-ws92w"] Nov 03 00:01:07.585350 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:01:07.585315 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:01:08.794736 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:08.794698 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a53ad7b3-45b4-4450-a3c7-b7e00596d28d" path="/var/lib/kubelet/pods/a53ad7b3-45b4-4450-a3c7-b7e00596d28d/volumes" Nov 03 00:01:10.345204 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.345164 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l"] Nov 03 00:01:10.345680 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.345605 5114 cpu_manager.go:414] "containerMap: removing container" podUID="53b68115-0b68-4d74-96ec-e70f18391969" containerName="cleanup" Nov 03 00:01:10.345680 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.345617 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b68115-0b68-4d74-96ec-e70f18391969" containerName="cleanup" Nov 03 00:01:10.345680 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.345637 5114 cpu_manager.go:414] "containerMap: removing container" podUID="a53ad7b3-45b4-4450-a3c7-b7e00596d28d" containerName="istio-proxy" Nov 03 00:01:10.345680 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.345642 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53ad7b3-45b4-4450-a3c7-b7e00596d28d" containerName="istio-proxy" Nov 03 00:01:10.345680 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.345657 5114 cpu_manager.go:414] "containerMap: removing container" podUID="89ca75ec-2516-4f65-b600-4b6d9c72908d" containerName="cleanup" Nov 03 00:01:10.345680 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.345663 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89ca75ec-2516-4f65-b600-4b6d9c72908d" containerName="cleanup" Nov 03 00:01:10.345680 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.345671 5114 cpu_manager.go:414] "containerMap: removing container" podUID="b0e0537f-9d0d-4179-8329-85218b5c9949" containerName="revoke-keys" Nov 03 00:01:10.345680 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.345678 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0e0537f-9d0d-4179-8329-85218b5c9949" containerName="revoke-keys" Nov 03 00:01:10.345938 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.345702 5114 cpu_manager.go:414] "containerMap: removing container" podUID="a53ad7b3-45b4-4450-a3c7-b7e00596d28d" containerName="istio-proxy" Nov 03 00:01:10.345938 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.345708 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53ad7b3-45b4-4450-a3c7-b7e00596d28d" containerName="istio-proxy" Nov 03 00:01:10.345938 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.345718 5114 cpu_manager.go:414] "containerMap: removing container" podUID="ccf010d7-6b97-4824-83da-9acac85d87bd" containerName="cleanup" Nov 03 00:01:10.345938 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.345723 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf010d7-6b97-4824-83da-9acac85d87bd" containerName="cleanup" Nov 03 00:01:10.345938 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.345794 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a294a5e-b315-4c9e-82f3-47f5dd2d729e" containerName="cleanup" Nov 03 00:01:10.345938 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.345804 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53ad7b3-45b4-4450-a3c7-b7e00596d28d" containerName="istio-proxy" Nov 03 00:01:10.345938 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.345811 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da46d94-2113-49ed-8349-d0a0f38cde59" containerName="cleanup" Nov 03 00:01:10.345938 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.345819 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b68115-0b68-4d74-96ec-e70f18391969" containerName="cleanup" Nov 03 00:01:10.345938 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.345824 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="89ca75ec-2516-4f65-b600-4b6d9c72908d" containerName="cleanup" Nov 03 00:01:10.345938 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.345830 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0e0537f-9d0d-4179-8329-85218b5c9949" containerName="revoke-keys" Nov 03 00:01:10.345938 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.345836 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53ad7b3-45b4-4450-a3c7-b7e00596d28d" containerName="istio-proxy" Nov 03 00:01:10.345938 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.345843 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="730f6d5e-c29d-491d-b40d-197ee83b76ff" containerName="cleanup" Nov 03 00:01:10.350196 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.350173 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:01:10.462155 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.462120 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-980b82\"/\"openshift-service-ca.crt\"" Nov 03 00:01:10.463390 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.463232 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-980b82\"/\"default-dockercfg-vqkvp\"" Nov 03 00:01:10.463390 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.463309 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-980b82\"/\"kube-root-ca.crt\"" Nov 03 00:01:10.463390 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.463362 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-980b82\"/\"rate-test-model-980b82-kserve-self-signed-certs\"" Nov 03 00:01:10.463390 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.462362 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvdfw\" (UniqueName: \"kubernetes.io/projected/6db69f57-95f7-4970-a795-0b31d9be79ed-kube-api-access-hvdfw\") pod \"rate-test-model-980b82-kserve-57679ddf77-pkc7l\" (UID: \"6db69f57-95f7-4970-a795-0b31d9be79ed\") " pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:01:10.463845 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.463426 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-dshm\") pod \"rate-test-model-980b82-kserve-57679ddf77-pkc7l\" (UID: \"6db69f57-95f7-4970-a795-0b31d9be79ed\") " pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:01:10.463845 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.463449 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db69f57-95f7-4970-a795-0b31d9be79ed-tls-certs\") pod \"rate-test-model-980b82-kserve-57679ddf77-pkc7l\" (UID: \"6db69f57-95f7-4970-a795-0b31d9be79ed\") " pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:01:10.463845 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.463597 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-model-cache\") pod \"rate-test-model-980b82-kserve-57679ddf77-pkc7l\" (UID: \"6db69f57-95f7-4970-a795-0b31d9be79ed\") " pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:01:10.463845 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.463643 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-tmp-dir\") pod \"rate-test-model-980b82-kserve-57679ddf77-pkc7l\" (UID: \"6db69f57-95f7-4970-a795-0b31d9be79ed\") " pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:01:10.463845 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.463701 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-home\") pod \"rate-test-model-980b82-kserve-57679ddf77-pkc7l\" (UID: \"6db69f57-95f7-4970-a795-0b31d9be79ed\") " pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:01:10.471987 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.471947 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l"] Nov 03 00:01:10.565508 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.565447 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-model-cache\") pod \"rate-test-model-980b82-kserve-57679ddf77-pkc7l\" (UID: \"6db69f57-95f7-4970-a795-0b31d9be79ed\") " pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:01:10.565731 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.565540 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-tmp-dir\") pod \"rate-test-model-980b82-kserve-57679ddf77-pkc7l\" (UID: \"6db69f57-95f7-4970-a795-0b31d9be79ed\") " pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:01:10.565731 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.565577 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-home\") pod \"rate-test-model-980b82-kserve-57679ddf77-pkc7l\" (UID: \"6db69f57-95f7-4970-a795-0b31d9be79ed\") " pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:01:10.565731 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.565659 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvdfw\" (UniqueName: \"kubernetes.io/projected/6db69f57-95f7-4970-a795-0b31d9be79ed-kube-api-access-hvdfw\") pod \"rate-test-model-980b82-kserve-57679ddf77-pkc7l\" (UID: \"6db69f57-95f7-4970-a795-0b31d9be79ed\") " pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:01:10.565731 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.565680 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-dshm\") pod \"rate-test-model-980b82-kserve-57679ddf77-pkc7l\" (UID: \"6db69f57-95f7-4970-a795-0b31d9be79ed\") " pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:01:10.565731 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.565695 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db69f57-95f7-4970-a795-0b31d9be79ed-tls-certs\") pod \"rate-test-model-980b82-kserve-57679ddf77-pkc7l\" (UID: \"6db69f57-95f7-4970-a795-0b31d9be79ed\") " pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:01:10.566000 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.565980 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-model-cache\") pod \"rate-test-model-980b82-kserve-57679ddf77-pkc7l\" (UID: \"6db69f57-95f7-4970-a795-0b31d9be79ed\") " pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:01:10.566000 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.565990 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-tmp-dir\") pod \"rate-test-model-980b82-kserve-57679ddf77-pkc7l\" (UID: \"6db69f57-95f7-4970-a795-0b31d9be79ed\") " pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:01:10.566141 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.566103 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-home\") pod \"rate-test-model-980b82-kserve-57679ddf77-pkc7l\" (UID: \"6db69f57-95f7-4970-a795-0b31d9be79ed\") " pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:01:10.568232 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.568205 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-dshm\") pod \"rate-test-model-980b82-kserve-57679ddf77-pkc7l\" (UID: \"6db69f57-95f7-4970-a795-0b31d9be79ed\") " pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:01:10.568523 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.568502 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db69f57-95f7-4970-a795-0b31d9be79ed-tls-certs\") pod \"rate-test-model-980b82-kserve-57679ddf77-pkc7l\" (UID: \"6db69f57-95f7-4970-a795-0b31d9be79ed\") " pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:01:10.629610 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.629529 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvdfw\" (UniqueName: \"kubernetes.io/projected/6db69f57-95f7-4970-a795-0b31d9be79ed-kube-api-access-hvdfw\") pod \"rate-test-model-980b82-kserve-57679ddf77-pkc7l\" (UID: \"6db69f57-95f7-4970-a795-0b31d9be79ed\") " pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:01:10.663184 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:10.663147 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:01:11.050471 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:11.050441 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l"] Nov 03 00:01:11.052305 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:01:11.052280 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6db69f57_95f7_4970_a795_0b31d9be79ed.slice/crio-60632f956e29b128066937155f03784f5582ca78577a44eeee1f8ec189d093e6 WatchSource:0}: Error finding container 60632f956e29b128066937155f03784f5582ca78577a44eeee1f8ec189d093e6: Status 404 returned error can't find the container with id 60632f956e29b128066937155f03784f5582ca78577a44eeee1f8ec189d093e6 Nov 03 00:01:12.004996 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:12.004953 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" event={"ID":"6db69f57-95f7-4970-a795-0b31d9be79ed","Type":"ContainerStarted","Data":"fbeac8c5bc4956307952b52fe3e8fd63e467f245fb123c458cdf439e63e102b3"} Nov 03 00:01:12.005411 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:12.005006 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" event={"ID":"6db69f57-95f7-4970-a795-0b31d9be79ed","Type":"ContainerStarted","Data":"60632f956e29b128066937155f03784f5582ca78577a44eeee1f8ec189d093e6"} Nov 03 00:01:12.120640 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:12.120586 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" podStartSLOduration=2.120564233 podStartE2EDuration="2.120564233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:01:12.119535636 +0000 UTC m=+4347.472569012" watchObservedRunningTime="2026-11-03 00:01:12.120564233 +0000 UTC m=+4347.473597621" Nov 03 00:01:12.663849 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:12.663810 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:01:20.663622 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:20.663582 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:01:20.676656 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:20.676625 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:01:21.132000 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:21.131970 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:01:37.584829 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:01:37.584791 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:01:46.647237 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:01:46.647200 5114 scope.go:122] "RemoveContainer" containerID="fb253bd8d0e3754c1b4c4628c86ca26e716844a9a9b68f99a141acd9cf11f13d" Nov 03 00:02:07.586460 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:02:07.586419 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:02:23.542176 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.542137 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7"] Nov 03 00:02:23.545875 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.545853 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:02:23.617994 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.617956 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-home\") pod \"rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7\" (UID: \"439f5c77-44d7-46b9-8b77-daa3df61b471\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:02:23.618209 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.618071 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-dshm\") pod \"rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7\" (UID: \"439f5c77-44d7-46b9-8b77-daa3df61b471\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:02:23.618209 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.618153 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-tmp-dir\") pod \"rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7\" (UID: \"439f5c77-44d7-46b9-8b77-daa3df61b471\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:02:23.618343 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.618316 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439f5c77-44d7-46b9-8b77-daa3df61b471-tls-certs\") pod \"rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7\" (UID: \"439f5c77-44d7-46b9-8b77-daa3df61b471\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:02:23.618407 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.618390 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvns4\" (UniqueName: \"kubernetes.io/projected/439f5c77-44d7-46b9-8b77-daa3df61b471-kube-api-access-xvns4\") pod \"rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7\" (UID: \"439f5c77-44d7-46b9-8b77-daa3df61b471\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:02:23.618458 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.618447 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-model-cache\") pod \"rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7\" (UID: \"439f5c77-44d7-46b9-8b77-daa3df61b471\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:02:23.647667 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.647627 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-b8d3cb\"/\"rate-test-model-b8d3cb-kserve-self-signed-certs\"" Nov 03 00:02:23.647667 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.647657 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-b8d3cb\"/\"openshift-service-ca.crt\"" Nov 03 00:02:23.647917 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.647703 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-b8d3cb\"/\"kube-root-ca.crt\"" Nov 03 00:02:23.647917 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.647712 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-b8d3cb\"/\"default-dockercfg-mhnqz\"" Nov 03 00:02:23.650251 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.650228 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7"] Nov 03 00:02:23.719247 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.719213 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-tmp-dir\") pod \"rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7\" (UID: \"439f5c77-44d7-46b9-8b77-daa3df61b471\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:02:23.719458 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.719294 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439f5c77-44d7-46b9-8b77-daa3df61b471-tls-certs\") pod \"rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7\" (UID: \"439f5c77-44d7-46b9-8b77-daa3df61b471\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:02:23.719458 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.719350 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvns4\" (UniqueName: \"kubernetes.io/projected/439f5c77-44d7-46b9-8b77-daa3df61b471-kube-api-access-xvns4\") pod \"rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7\" (UID: \"439f5c77-44d7-46b9-8b77-daa3df61b471\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:02:23.719458 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.719400 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-model-cache\") pod \"rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7\" (UID: \"439f5c77-44d7-46b9-8b77-daa3df61b471\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:02:23.719670 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.719516 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-home\") pod \"rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7\" (UID: \"439f5c77-44d7-46b9-8b77-daa3df61b471\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:02:23.719670 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.719572 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-dshm\") pod \"rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7\" (UID: \"439f5c77-44d7-46b9-8b77-daa3df61b471\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:02:23.719793 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.719690 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-tmp-dir\") pod \"rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7\" (UID: \"439f5c77-44d7-46b9-8b77-daa3df61b471\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:02:23.719920 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.719897 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-home\") pod \"rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7\" (UID: \"439f5c77-44d7-46b9-8b77-daa3df61b471\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:02:23.720239 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.720213 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-model-cache\") pod \"rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7\" (UID: \"439f5c77-44d7-46b9-8b77-daa3df61b471\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:02:23.722453 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.722421 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-dshm\") pod \"rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7\" (UID: \"439f5c77-44d7-46b9-8b77-daa3df61b471\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:02:23.722901 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.722875 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439f5c77-44d7-46b9-8b77-daa3df61b471-tls-certs\") pod \"rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7\" (UID: \"439f5c77-44d7-46b9-8b77-daa3df61b471\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:02:23.780815 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.780777 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvns4\" (UniqueName: \"kubernetes.io/projected/439f5c77-44d7-46b9-8b77-daa3df61b471-kube-api-access-xvns4\") pod \"rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7\" (UID: \"439f5c77-44d7-46b9-8b77-daa3df61b471\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:02:23.858892 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:23.858799 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:02:24.245421 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:24.245393 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7"] Nov 03 00:02:24.247204 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:02:24.247177 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod439f5c77_44d7_46b9_8b77_daa3df61b471.slice/crio-3341d2f98a250c438952107cf4bdc6b364fa25b92730297ded425e0b15e4a534 WatchSource:0}: Error finding container 3341d2f98a250c438952107cf4bdc6b364fa25b92730297ded425e0b15e4a534: Status 404 returned error can't find the container with id 3341d2f98a250c438952107cf4bdc6b364fa25b92730297ded425e0b15e4a534 Nov 03 00:02:24.480624 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:24.480589 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" event={"ID":"439f5c77-44d7-46b9-8b77-daa3df61b471","Type":"ContainerStarted","Data":"b64495ed8beb13a3033b4eca61e22789d699a6723c282a378b81ce1b50507f24"} Nov 03 00:02:24.480820 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:24.480637 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" event={"ID":"439f5c77-44d7-46b9-8b77-daa3df61b471","Type":"ContainerStarted","Data":"3341d2f98a250c438952107cf4bdc6b364fa25b92730297ded425e0b15e4a534"} Nov 03 00:02:24.597350 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:24.597288 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" podStartSLOduration=1.597272086 podStartE2EDuration="1.597272086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:02:24.596304021 +0000 UTC m=+4419.949337397" watchObservedRunningTime="2026-11-03 00:02:24.597272086 +0000 UTC m=+4419.950305518" Nov 03 00:02:24.859890 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:24.859795 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:02:33.859185 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:33.859137 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:02:33.872176 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:33.872143 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:02:34.546890 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:34.546861 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:02:35.567863 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.567823 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8"] Nov 03 00:02:35.572098 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.572072 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.624387 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.624344 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-04713b77-openshift-default-dockercfg-qppsl\"" Nov 03 00:02:35.666988 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.666951 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-istio-envoy\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.666988 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.666994 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-credential-socket\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.667283 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.667017 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsh49\" (UniqueName: \"kubernetes.io/projected/03ca2528-0901-4893-9d53-2705cece0aee-kube-api-access-jsh49\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.667283 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.667189 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-workload-socket\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.667283 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.667260 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-istio-data\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.667452 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.667288 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ca2528-0901-4893-9d53-2705cece0aee-istiod-ca-cert\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.667452 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.667441 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ca2528-0901-4893-9d53-2705cece0aee-istio-podinfo\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.667603 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.667473 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ca2528-0901-4893-9d53-2705cece0aee-istio-token\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.667603 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.667558 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-workload-certs\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.676419 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.676384 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8"] Nov 03 00:02:35.773032 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.772237 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ca2528-0901-4893-9d53-2705cece0aee-istio-podinfo\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.773032 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.772299 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ca2528-0901-4893-9d53-2705cece0aee-istio-token\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.773032 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.772349 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-workload-certs\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.773032 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.772473 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-istio-envoy\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.773032 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.772525 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-credential-socket\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.773032 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.772560 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsh49\" (UniqueName: \"kubernetes.io/projected/03ca2528-0901-4893-9d53-2705cece0aee-kube-api-access-jsh49\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.773032 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.772609 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-workload-socket\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.773032 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.772642 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-istio-data\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.773032 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.772668 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ca2528-0901-4893-9d53-2705cece0aee-istiod-ca-cert\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.782842 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.773886 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ca2528-0901-4893-9d53-2705cece0aee-istiod-ca-cert\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.782842 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.774245 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-workload-certs\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.782842 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.775190 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-credential-socket\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.782842 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.775454 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-workload-socket\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.782842 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.775711 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-istio-data\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.782842 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.778737 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-istio-envoy\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.782842 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.781285 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ca2528-0901-4893-9d53-2705cece0aee-istio-podinfo\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.833593 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.833475 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ca2528-0901-4893-9d53-2705cece0aee-istio-token\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.835382 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.835350 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsh49\" (UniqueName: \"kubernetes.io/projected/03ca2528-0901-4893-9d53-2705cece0aee-kube-api-access-jsh49\") pod \"e2e-mt-04713b77-openshift-default-866b79f589-ljbr8\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:35.887255 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:35.887209 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:36.319290 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:36.319262 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8"] Nov 03 00:02:36.322659 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:02:36.322623 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ca2528_0901_4893_9d53_2705cece0aee.slice/crio-0bb0641e9abadb6dad223b91984b1218569047c70dbf24207813f7cf4b4e1c1a WatchSource:0}: Error finding container 0bb0641e9abadb6dad223b91984b1218569047c70dbf24207813f7cf4b4e1c1a: Status 404 returned error can't find the container with id 0bb0641e9abadb6dad223b91984b1218569047c70dbf24207813f7cf4b4e1c1a Nov 03 00:02:36.325980 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:36.325937 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:02:36.326115 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:36.326028 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:02:36.326115 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:36.326077 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:02:36.569448 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:36.569355 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" event={"ID":"03ca2528-0901-4893-9d53-2705cece0aee","Type":"ContainerStarted","Data":"f6679ff2142673a7fb4398d1df8e39248230e5ade3549738378dbc984eeb51ed"} Nov 03 00:02:36.569448 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:36.569416 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" event={"ID":"03ca2528-0901-4893-9d53-2705cece0aee","Type":"ContainerStarted","Data":"0bb0641e9abadb6dad223b91984b1218569047c70dbf24207813f7cf4b4e1c1a"} Nov 03 00:02:36.887697 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:36.887592 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:37.583969 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:02:37.583926 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:02:37.894430 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:37.894341 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:38.114343 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:38.114281 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" podStartSLOduration=3.114265919 podStartE2EDuration="3.114265919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:02:36.695680881 +0000 UTC m=+4432.048714258" watchObservedRunningTime="2026-11-03 00:02:38.114265919 +0000 UTC m=+4433.467299294" Nov 03 00:02:38.580370 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:38.580337 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:02:38.581391 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:02:38.581368 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:03:07.584300 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:03:07.584263 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:03:37.585140 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:03:37.585094 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:04:07.585087 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:04:07.585044 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:04:15.047493 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:15.047452 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5feec3c6-z8vbl"] Nov 03 00:04:15.096553 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:15.096513 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5feec3c6-z8vbl"] Nov 03 00:04:16.794085 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:16.794052 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7da2e976-6a1b-4ba5-8271-e10189770d6a" path="/var/lib/kubelet/pods/7da2e976-6a1b-4ba5-8271-e10189770d6a/volumes" Nov 03 00:04:20.228545 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:20.228503 5114 generic.go:356] "Generic (PLEG): container finished" podID="03ca2528-0901-4893-9d53-2705cece0aee" containerID="f6679ff2142673a7fb4398d1df8e39248230e5ade3549738378dbc984eeb51ed" exitCode=0 Nov 03 00:04:20.229050 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:20.228591 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" event={"ID":"03ca2528-0901-4893-9d53-2705cece0aee","Type":"ContainerDied","Data":"f6679ff2142673a7fb4398d1df8e39248230e5ade3549738378dbc984eeb51ed"} Nov 03 00:04:20.229097 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:20.229051 5114 scope.go:122] "RemoveContainer" containerID="f6679ff2142673a7fb4398d1df8e39248230e5ade3549738378dbc984eeb51ed" Nov 03 00:04:20.230706 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:20.230677 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:04:20.230793 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:20.230753 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:04:20.230793 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:20.230789 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:04:21.238102 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:21.238058 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" event={"ID":"03ca2528-0901-4893-9d53-2705cece0aee","Type":"ContainerStarted","Data":"661b6be539b8f227888754e0ab4727badc6c15256df029bdad084c8514e40e62"} Nov 03 00:04:21.888274 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:21.888232 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:04:21.889773 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:21.889742 5114 patch_prober.go:29] interesting pod/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 03 00:04:21.889913 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:21.889810 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" podUID="03ca2528-0901-4893-9d53-2705cece0aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 03 00:04:22.887684 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:22.887640 5114 patch_prober.go:29] interesting pod/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 03 00:04:22.888098 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:22.887698 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" podUID="03ca2528-0901-4893-9d53-2705cece0aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 03 00:04:23.580803 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:23.580772 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:04:23.887647 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:23.887560 5114 patch_prober.go:29] interesting pod/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 03 00:04:23.887647 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:23.887625 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" podUID="03ca2528-0901-4893-9d53-2705cece0aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 03 00:04:24.887771 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:24.887731 5114 patch_prober.go:29] interesting pod/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 03 00:04:24.887972 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:24.887795 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" podUID="03ca2528-0901-4893-9d53-2705cece0aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 03 00:04:25.888318 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:25.888273 5114 patch_prober.go:29] interesting pod/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 03 00:04:25.888846 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:25.888338 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" podUID="03ca2528-0901-4893-9d53-2705cece0aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 03 00:04:26.892768 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:26.892732 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:04:26.893944 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:26.893921 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:04:37.585847 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:04:37.585806 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:04:53.046820 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:53.046780 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f5da0f1-8q8fm"] Nov 03 00:04:53.096740 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:53.096699 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f5da0f1-8q8fm"] Nov 03 00:04:54.794230 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:04:54.794184 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1afacc90-2b81-42c2-86af-1622afdfca74" path="/var/lib/kubelet/pods/1afacc90-2b81-42c2-86af-1622afdfca74/volumes" Nov 03 00:05:07.584720 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:05:07.584674 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:05:29.268588 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:29.268463 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh"] Nov 03 00:05:29.269200 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:29.268800 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" podUID="55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" containerName="istio-proxy" containerID="cri-o://e7bfd35221ff1a530f36ca03d0067cabf49c05ce34598bc44414d286ba20cd43" gracePeriod=30 Nov 03 00:05:30.357705 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:30.357659 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-726e8bf6-dqtdj"] Nov 03 00:05:30.360353 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:30.360329 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-726e8bf6-dqtdj" Nov 03 00:05:30.412397 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:30.412364 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 03 00:05:30.461913 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:30.461870 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:05:30.464517 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:30.464468 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-726e8bf6-dqtdj"] Nov 03 00:05:30.487521 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:30.487440 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c95w\" (UniqueName: \"kubernetes.io/projected/9f78f19e-8722-4269-919c-289f0da197fa-kube-api-access-8c95w\") pod \"maas-api-revoke-sub-726e8bf6-dqtdj\" (UID: \"9f78f19e-8722-4269-919c-289f0da197fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726e8bf6-dqtdj" Nov 03 00:05:30.487722 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:30.487599 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f78f19e-8722-4269-919c-289f0da197fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-726e8bf6-dqtdj\" (UID: \"9f78f19e-8722-4269-919c-289f0da197fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726e8bf6-dqtdj" Nov 03 00:05:30.487722 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:30.487648 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f78f19e-8722-4269-919c-289f0da197fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-726e8bf6-dqtdj\" (UID: \"9f78f19e-8722-4269-919c-289f0da197fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726e8bf6-dqtdj" Nov 03 00:05:30.588805 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:30.588762 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c95w\" (UniqueName: \"kubernetes.io/projected/9f78f19e-8722-4269-919c-289f0da197fa-kube-api-access-8c95w\") pod \"maas-api-revoke-sub-726e8bf6-dqtdj\" (UID: \"9f78f19e-8722-4269-919c-289f0da197fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726e8bf6-dqtdj" Nov 03 00:05:30.589040 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:30.588826 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f78f19e-8722-4269-919c-289f0da197fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-726e8bf6-dqtdj\" (UID: \"9f78f19e-8722-4269-919c-289f0da197fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726e8bf6-dqtdj" Nov 03 00:05:30.589040 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:30.588895 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f78f19e-8722-4269-919c-289f0da197fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-726e8bf6-dqtdj\" (UID: \"9f78f19e-8722-4269-919c-289f0da197fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726e8bf6-dqtdj" Nov 03 00:05:30.589650 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:30.589630 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f78f19e-8722-4269-919c-289f0da197fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-726e8bf6-dqtdj\" (UID: \"9f78f19e-8722-4269-919c-289f0da197fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726e8bf6-dqtdj" Nov 03 00:05:30.591524 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:30.591476 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f78f19e-8722-4269-919c-289f0da197fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-726e8bf6-dqtdj\" (UID: \"9f78f19e-8722-4269-919c-289f0da197fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726e8bf6-dqtdj" Nov 03 00:05:30.649591 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:30.649510 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c95w\" (UniqueName: \"kubernetes.io/projected/9f78f19e-8722-4269-919c-289f0da197fa-kube-api-access-8c95w\") pod \"maas-api-revoke-sub-726e8bf6-dqtdj\" (UID: \"9f78f19e-8722-4269-919c-289f0da197fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726e8bf6-dqtdj" Nov 03 00:05:30.673704 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:30.673664 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-726e8bf6-dqtdj" Nov 03 00:05:31.052436 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:31.052406 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-726e8bf6-dqtdj"] Nov 03 00:05:31.054092 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:05:31.054056 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f78f19e_8722_4269_919c_289f0da197fa.slice/crio-ffff25ddebd4c71a5774ff79a56860eed1055f1a3b2717aba799adedf9bceddf WatchSource:0}: Error finding container ffff25ddebd4c71a5774ff79a56860eed1055f1a3b2717aba799adedf9bceddf: Status 404 returned error can't find the container with id ffff25ddebd4c71a5774ff79a56860eed1055f1a3b2717aba799adedf9bceddf Nov 03 00:05:31.120677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:31.120609 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f039fc5-lc96x"] Nov 03 00:05:31.123929 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:31.123901 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f039fc5-lc96x" Nov 03 00:05:31.182258 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:31.182221 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f039fc5-lc96x"] Nov 03 00:05:31.196202 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:31.196158 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z24r\" (UniqueName: \"kubernetes.io/projected/cbb7a506-9811-4530-bde6-ccb4bc2b64da-kube-api-access-5z24r\") pod \"maas-api-revoke-sub-6f039fc5-lc96x\" (UID: \"cbb7a506-9811-4530-bde6-ccb4bc2b64da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f039fc5-lc96x" Nov 03 00:05:31.196376 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:31.196276 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbb7a506-9811-4530-bde6-ccb4bc2b64da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f039fc5-lc96x\" (UID: \"cbb7a506-9811-4530-bde6-ccb4bc2b64da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f039fc5-lc96x" Nov 03 00:05:31.196454 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:31.196435 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbb7a506-9811-4530-bde6-ccb4bc2b64da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f039fc5-lc96x\" (UID: \"cbb7a506-9811-4530-bde6-ccb4bc2b64da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f039fc5-lc96x" Nov 03 00:05:31.297836 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:31.297795 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbb7a506-9811-4530-bde6-ccb4bc2b64da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f039fc5-lc96x\" (UID: \"cbb7a506-9811-4530-bde6-ccb4bc2b64da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f039fc5-lc96x" Nov 03 00:05:31.298046 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:31.297915 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z24r\" (UniqueName: \"kubernetes.io/projected/cbb7a506-9811-4530-bde6-ccb4bc2b64da-kube-api-access-5z24r\") pod \"maas-api-revoke-sub-6f039fc5-lc96x\" (UID: \"cbb7a506-9811-4530-bde6-ccb4bc2b64da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f039fc5-lc96x" Nov 03 00:05:31.298046 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:31.297948 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbb7a506-9811-4530-bde6-ccb4bc2b64da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f039fc5-lc96x\" (UID: \"cbb7a506-9811-4530-bde6-ccb4bc2b64da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f039fc5-lc96x" Nov 03 00:05:31.298612 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:31.298578 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbb7a506-9811-4530-bde6-ccb4bc2b64da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f039fc5-lc96x\" (UID: \"cbb7a506-9811-4530-bde6-ccb4bc2b64da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f039fc5-lc96x" Nov 03 00:05:31.300808 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:31.300752 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbb7a506-9811-4530-bde6-ccb4bc2b64da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f039fc5-lc96x\" (UID: \"cbb7a506-9811-4530-bde6-ccb4bc2b64da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f039fc5-lc96x" Nov 03 00:05:31.358674 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:31.358638 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z24r\" (UniqueName: \"kubernetes.io/projected/cbb7a506-9811-4530-bde6-ccb4bc2b64da-kube-api-access-5z24r\") pod \"maas-api-revoke-sub-6f039fc5-lc96x\" (UID: \"cbb7a506-9811-4530-bde6-ccb4bc2b64da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f039fc5-lc96x" Nov 03 00:05:31.437081 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:31.437038 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f039fc5-lc96x" Nov 03 00:05:31.655517 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:31.655392 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-726e8bf6-dqtdj" event={"ID":"9f78f19e-8722-4269-919c-289f0da197fa","Type":"ContainerStarted","Data":"dd30f00b923386937322e4a60dce2d7b3013f33e5d930ac0355414028a39270c"} Nov 03 00:05:31.655517 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:31.655448 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-726e8bf6-dqtdj" event={"ID":"9f78f19e-8722-4269-919c-289f0da197fa","Type":"ContainerStarted","Data":"ffff25ddebd4c71a5774ff79a56860eed1055f1a3b2717aba799adedf9bceddf"} Nov 03 00:05:31.773267 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:31.773210 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-726e8bf6-dqtdj" podStartSLOduration=1.7731935810000001 podStartE2EDuration="1.773193581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:31.770818317 +0000 UTC m=+4607.123851693" watchObservedRunningTime="2026-11-03 00:05:31.773193581 +0000 UTC m=+4607.126226960" Nov 03 00:05:31.811592 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:31.811567 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f039fc5-lc96x"] Nov 03 00:05:31.813497 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:05:31.813447 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb7a506_9811_4530_bde6_ccb4bc2b64da.slice/crio-8ab9f498635a1b6924e2e17e23c75f85301c25d93bb0ea4ac46b7a16835b71bb WatchSource:0}: Error finding container 8ab9f498635a1b6924e2e17e23c75f85301c25d93bb0ea4ac46b7a16835b71bb: Status 404 returned error can't find the container with id 8ab9f498635a1b6924e2e17e23c75f85301c25d93bb0ea4ac46b7a16835b71bb Nov 03 00:05:32.663824 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:32.663788 5114 generic.go:356] "Generic (PLEG): container finished" podID="9f78f19e-8722-4269-919c-289f0da197fa" containerID="dd30f00b923386937322e4a60dce2d7b3013f33e5d930ac0355414028a39270c" exitCode=0 Nov 03 00:05:32.664532 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:32.663876 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-726e8bf6-dqtdj" event={"ID":"9f78f19e-8722-4269-919c-289f0da197fa","Type":"ContainerDied","Data":"dd30f00b923386937322e4a60dce2d7b3013f33e5d930ac0355414028a39270c"} Nov 03 00:05:32.665677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:32.665649 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f039fc5-lc96x" event={"ID":"cbb7a506-9811-4530-bde6-ccb4bc2b64da","Type":"ContainerStarted","Data":"9673699b578a19dd536ca8c702c4dc749a0cf2c11b71be546dbd578a67d70369"} Nov 03 00:05:32.665832 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:32.665704 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f039fc5-lc96x" event={"ID":"cbb7a506-9811-4530-bde6-ccb4bc2b64da","Type":"ContainerStarted","Data":"8ab9f498635a1b6924e2e17e23c75f85301c25d93bb0ea4ac46b7a16835b71bb"} Nov 03 00:05:32.821585 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:32.821547 5114 patch_prober.go:29] interesting pod/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:05:32.821728 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:32.821617 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" podUID="55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:05:32.904564 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:32.904509 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f039fc5-lc96x" podStartSLOduration=1.90447359 podStartE2EDuration="1.90447359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:32.903007921 +0000 UTC m=+4608.256041297" watchObservedRunningTime="2026-11-03 00:05:32.90447359 +0000 UTC m=+4608.257506964" Nov 03 00:05:33.673529 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:33.673457 5114 generic.go:356] "Generic (PLEG): container finished" podID="cbb7a506-9811-4530-bde6-ccb4bc2b64da" containerID="9673699b578a19dd536ca8c702c4dc749a0cf2c11b71be546dbd578a67d70369" exitCode=0 Nov 03 00:05:33.674055 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:33.673541 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f039fc5-lc96x" event={"ID":"cbb7a506-9811-4530-bde6-ccb4bc2b64da","Type":"ContainerDied","Data":"9673699b578a19dd536ca8c702c4dc749a0cf2c11b71be546dbd578a67d70369"} Nov 03 00:05:33.846716 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:33.846686 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-726e8bf6-dqtdj" Nov 03 00:05:33.931302 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:33.931210 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f78f19e-8722-4269-919c-289f0da197fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f78f19e-8722-4269-919c-289f0da197fa-maas-api-xks-ca\") pod \"9f78f19e-8722-4269-919c-289f0da197fa\" (UID: \"9f78f19e-8722-4269-919c-289f0da197fa\") " Nov 03 00:05:33.931534 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:33.931323 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f78f19e-8722-4269-919c-289f0da197fa-kube-api-access-8c95w\" (UniqueName: \"kubernetes.io/projected/9f78f19e-8722-4269-919c-289f0da197fa-kube-api-access-8c95w\") pod \"9f78f19e-8722-4269-919c-289f0da197fa\" (UID: \"9f78f19e-8722-4269-919c-289f0da197fa\") " Nov 03 00:05:33.931534 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:33.931346 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f78f19e-8722-4269-919c-289f0da197fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f78f19e-8722-4269-919c-289f0da197fa-maas-api-openshift-ca\") pod \"9f78f19e-8722-4269-919c-289f0da197fa\" (UID: \"9f78f19e-8722-4269-919c-289f0da197fa\") " Nov 03 00:05:33.932391 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:33.931864 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f78f19e-8722-4269-919c-289f0da197fa-maas-api-openshift-ca" pod "9f78f19e-8722-4269-919c-289f0da197fa" (UID: "9f78f19e-8722-4269-919c-289f0da197fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:05:33.932391 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:33.932025 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f78f19e-8722-4269-919c-289f0da197fa-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:33.933567 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:33.933546 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f78f19e-8722-4269-919c-289f0da197fa-maas-api-xks-ca" pod "9f78f19e-8722-4269-919c-289f0da197fa" (UID: "9f78f19e-8722-4269-919c-289f0da197fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:05:33.933889 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:33.933855 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f78f19e-8722-4269-919c-289f0da197fa-kube-api-access-8c95w" pod "9f78f19e-8722-4269-919c-289f0da197fa" (UID: "9f78f19e-8722-4269-919c-289f0da197fa"). InnerVolumeSpecName "kube-api-access-8c95w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:34.033701 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.033657 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c95w\" (UniqueName: \"kubernetes.io/projected/9f78f19e-8722-4269-919c-289f0da197fa-kube-api-access-8c95w\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:34.033701 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.033694 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f78f19e-8722-4269-919c-289f0da197fa-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:34.505368 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.505339 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 03 00:05:34.639809 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.639776 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-envoy\") pod \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " Nov 03 00:05:34.639984 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.639864 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-workload-socket\") pod \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " Nov 03 00:05:34.639984 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.639949 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-kube-api-access-68ndw\" (UniqueName: \"kubernetes.io/projected/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-kube-api-access-68ndw\") pod \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " Nov 03 00:05:34.640117 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.639985 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-workload-certs\") pod \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " Nov 03 00:05:34.640117 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.640032 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-data\") pod \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " Nov 03 00:05:34.640117 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.640065 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-credential-socket\") pod \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " Nov 03 00:05:34.640117 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.640108 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-podinfo\") pod \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " Nov 03 00:05:34.640332 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.640157 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-workload-socket" pod "55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" (UID: "55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:34.640332 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.640190 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istiod-ca-cert\") pod \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " Nov 03 00:05:34.640332 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.640257 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-token\" (UniqueName: \"kubernetes.io/projected/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-token\") pod \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\" (UID: \"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046\") " Nov 03 00:05:34.640332 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.640274 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-workload-certs" pod "55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" (UID: "55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:34.640586 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.640352 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-data" pod "55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" (UID: "55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:34.640586 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.640373 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-credential-socket" pod "55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" (UID: "55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:34.640722 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.640704 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istiod-ca-cert" pod "55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" (UID: "55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:05:34.641018 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.640950 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:34.641018 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.640989 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:34.641018 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.641007 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:34.641018 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.641022 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:34.641279 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.641037 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:34.642305 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.642278 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-envoy" pod "55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" (UID: "55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:34.642451 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.642429 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-kube-api-access-68ndw" pod "55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" (UID: "55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046"). InnerVolumeSpecName "kube-api-access-68ndw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:34.642543 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.642512 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-podinfo" pod "55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" (UID: "55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:05:34.642848 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.642830 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-token" pod "55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" (UID: "55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:34.680623 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.680587 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv"] Nov 03 00:05:34.685284 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.685251 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-726e8bf6-dqtdj" Nov 03 00:05:34.685505 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.685280 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-726e8bf6-dqtdj" event={"ID":"9f78f19e-8722-4269-919c-289f0da197fa","Type":"ContainerDied","Data":"ffff25ddebd4c71a5774ff79a56860eed1055f1a3b2717aba799adedf9bceddf"} Nov 03 00:05:34.685505 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.685349 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffff25ddebd4c71a5774ff79a56860eed1055f1a3b2717aba799adedf9bceddf" Nov 03 00:05:34.688503 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.688456 5114 generic.go:356] "Generic (PLEG): container finished" podID="55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" containerID="e7bfd35221ff1a530f36ca03d0067cabf49c05ce34598bc44414d286ba20cd43" exitCode=0 Nov 03 00:05:34.688666 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.688605 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" Nov 03 00:05:34.688747 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.688658 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" event={"ID":"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046","Type":"ContainerDied","Data":"e7bfd35221ff1a530f36ca03d0067cabf49c05ce34598bc44414d286ba20cd43"} Nov 03 00:05:34.688747 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.688709 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh" event={"ID":"55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046","Type":"ContainerDied","Data":"c2ce116c4e1aa3f237d18c88b35e90bd1e8f0f7a120fcf792aef5611fafae206"} Nov 03 00:05:34.688747 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.688740 5114 scope.go:122] "RemoveContainer" containerID="e7bfd35221ff1a530f36ca03d0067cabf49c05ce34598bc44414d286ba20cd43" Nov 03 00:05:34.691715 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.691664 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt"] Nov 03 00:05:34.725228 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.725198 5114 scope.go:122] "RemoveContainer" containerID="3800a86b7c6190d0fb87f7bef48260406279358468ecad3354f8cafaaf7286bb" Nov 03 00:05:34.738715 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.738688 5114 scope.go:122] "RemoveContainer" containerID="e7bfd35221ff1a530f36ca03d0067cabf49c05ce34598bc44414d286ba20cd43" Nov 03 00:05:34.739077 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:05:34.739055 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7bfd35221ff1a530f36ca03d0067cabf49c05ce34598bc44414d286ba20cd43\": container with ID starting with e7bfd35221ff1a530f36ca03d0067cabf49c05ce34598bc44414d286ba20cd43 not found: ID does not exist" containerID="e7bfd35221ff1a530f36ca03d0067cabf49c05ce34598bc44414d286ba20cd43" Nov 03 00:05:34.739153 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.739087 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7bfd35221ff1a530f36ca03d0067cabf49c05ce34598bc44414d286ba20cd43"} err="failed to get container status \"e7bfd35221ff1a530f36ca03d0067cabf49c05ce34598bc44414d286ba20cd43\": rpc error: code = NotFound desc = could not find container \"e7bfd35221ff1a530f36ca03d0067cabf49c05ce34598bc44414d286ba20cd43\": container with ID starting with e7bfd35221ff1a530f36ca03d0067cabf49c05ce34598bc44414d286ba20cd43 not found: ID does not exist" Nov 03 00:05:34.739153 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.739108 5114 scope.go:122] "RemoveContainer" containerID="3800a86b7c6190d0fb87f7bef48260406279358468ecad3354f8cafaaf7286bb" Nov 03 00:05:34.739383 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:05:34.739355 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3800a86b7c6190d0fb87f7bef48260406279358468ecad3354f8cafaaf7286bb\": container with ID starting with 3800a86b7c6190d0fb87f7bef48260406279358468ecad3354f8cafaaf7286bb not found: ID does not exist" containerID="3800a86b7c6190d0fb87f7bef48260406279358468ecad3354f8cafaaf7286bb" Nov 03 00:05:34.739383 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.739377 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3800a86b7c6190d0fb87f7bef48260406279358468ecad3354f8cafaaf7286bb"} err="failed to get container status \"3800a86b7c6190d0fb87f7bef48260406279358468ecad3354f8cafaaf7286bb\": rpc error: code = NotFound desc = could not find container \"3800a86b7c6190d0fb87f7bef48260406279358468ecad3354f8cafaaf7286bb\": container with ID starting with 3800a86b7c6190d0fb87f7bef48260406279358468ecad3354f8cafaaf7286bb not found: ID does not exist" Nov 03 00:05:34.742126 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.742100 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:34.742126 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.742127 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:34.742276 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.742144 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68ndw\" (UniqueName: \"kubernetes.io/projected/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-kube-api-access-68ndw\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:34.742276 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.742154 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:34.868015 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:34.867983 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f039fc5-lc96x" Nov 03 00:05:35.045804 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:35.045754 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbb7a506-9811-4530-bde6-ccb4bc2b64da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbb7a506-9811-4530-bde6-ccb4bc2b64da-maas-api-xks-ca\") pod \"cbb7a506-9811-4530-bde6-ccb4bc2b64da\" (UID: \"cbb7a506-9811-4530-bde6-ccb4bc2b64da\") " Nov 03 00:05:35.045999 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:35.045828 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbb7a506-9811-4530-bde6-ccb4bc2b64da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbb7a506-9811-4530-bde6-ccb4bc2b64da-maas-api-openshift-ca\") pod \"cbb7a506-9811-4530-bde6-ccb4bc2b64da\" (UID: \"cbb7a506-9811-4530-bde6-ccb4bc2b64da\") " Nov 03 00:05:35.046086 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:35.046068 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb7a506-9811-4530-bde6-ccb4bc2b64da-kube-api-access-5z24r\" (UniqueName: \"kubernetes.io/projected/cbb7a506-9811-4530-bde6-ccb4bc2b64da-kube-api-access-5z24r\") pod \"cbb7a506-9811-4530-bde6-ccb4bc2b64da\" (UID: \"cbb7a506-9811-4530-bde6-ccb4bc2b64da\") " Nov 03 00:05:35.046216 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:35.046188 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb7a506-9811-4530-bde6-ccb4bc2b64da-maas-api-openshift-ca" pod "cbb7a506-9811-4530-bde6-ccb4bc2b64da" (UID: "cbb7a506-9811-4530-bde6-ccb4bc2b64da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:05:35.046567 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:35.046530 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbb7a506-9811-4530-bde6-ccb4bc2b64da-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:35.048384 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:35.048355 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbb7a506-9811-4530-bde6-ccb4bc2b64da-maas-api-xks-ca" pod "cbb7a506-9811-4530-bde6-ccb4bc2b64da" (UID: "cbb7a506-9811-4530-bde6-ccb4bc2b64da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:05:35.048506 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:35.048411 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb7a506-9811-4530-bde6-ccb4bc2b64da-kube-api-access-5z24r" pod "cbb7a506-9811-4530-bde6-ccb4bc2b64da" (UID: "cbb7a506-9811-4530-bde6-ccb4bc2b64da"). InnerVolumeSpecName "kube-api-access-5z24r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:35.147972 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:35.147936 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z24r\" (UniqueName: \"kubernetes.io/projected/cbb7a506-9811-4530-bde6-ccb4bc2b64da-kube-api-access-5z24r\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:35.147972 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:35.147971 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbb7a506-9811-4530-bde6-ccb4bc2b64da-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:35.253314 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:35.253271 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh"] Nov 03 00:05:35.258554 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:35.258519 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-915ccb-openshift-default-6cfd7f7575-ffpvh"] Nov 03 00:05:35.695423 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:35.695389 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f039fc5-lc96x" Nov 03 00:05:35.695957 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:35.695425 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f039fc5-lc96x" event={"ID":"cbb7a506-9811-4530-bde6-ccb4bc2b64da","Type":"ContainerDied","Data":"8ab9f498635a1b6924e2e17e23c75f85301c25d93bb0ea4ac46b7a16835b71bb"} Nov 03 00:05:35.695957 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:35.695519 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ab9f498635a1b6924e2e17e23c75f85301c25d93bb0ea4ac46b7a16835b71bb" Nov 03 00:05:36.794459 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:36.794418 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" path="/var/lib/kubelet/pods/55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046/volumes" Nov 03 00:05:37.584918 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:05:37.584880 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:05:41.807526 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:41.807458 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0a1c13d-xcv5j"] Nov 03 00:05:41.807992 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:41.807930 5114 cpu_manager.go:414] "containerMap: removing container" podUID="cbb7a506-9811-4530-bde6-ccb4bc2b64da" containerName="revoke-keys" Nov 03 00:05:41.807992 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:41.807943 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb7a506-9811-4530-bde6-ccb4bc2b64da" containerName="revoke-keys" Nov 03 00:05:41.807992 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:41.807957 5114 cpu_manager.go:414] "containerMap: removing container" podUID="55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" containerName="istio-proxy" Nov 03 00:05:41.807992 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:41.807962 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" containerName="istio-proxy" Nov 03 00:05:41.807992 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:41.807980 5114 cpu_manager.go:414] "containerMap: removing container" podUID="9f78f19e-8722-4269-919c-289f0da197fa" containerName="revoke-keys" Nov 03 00:05:41.807992 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:41.807985 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f78f19e-8722-4269-919c-289f0da197fa" containerName="revoke-keys" Nov 03 00:05:41.808193 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:41.808068 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb7a506-9811-4530-bde6-ccb4bc2b64da" containerName="revoke-keys" Nov 03 00:05:41.808193 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:41.808078 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" containerName="istio-proxy" Nov 03 00:05:41.808193 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:41.808087 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f78f19e-8722-4269-919c-289f0da197fa" containerName="revoke-keys" Nov 03 00:05:41.810582 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:41.810562 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a1c13d-xcv5j" Nov 03 00:05:41.825231 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:41.825191 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ec560e-262c-4f97-836e-ccdaeb231102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0a1c13d-xcv5j\" (UID: \"65ec560e-262c-4f97-836e-ccdaeb231102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a1c13d-xcv5j" Nov 03 00:05:41.825384 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:41.825258 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ec560e-262c-4f97-836e-ccdaeb231102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0a1c13d-xcv5j\" (UID: \"65ec560e-262c-4f97-836e-ccdaeb231102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a1c13d-xcv5j" Nov 03 00:05:41.825660 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:41.825620 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkp2v\" (UniqueName: \"kubernetes.io/projected/65ec560e-262c-4f97-836e-ccdaeb231102-kube-api-access-jkp2v\") pod \"maas-api-revoke-sub-a0a1c13d-xcv5j\" (UID: \"65ec560e-262c-4f97-836e-ccdaeb231102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a1c13d-xcv5j" Nov 03 00:05:41.862698 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:41.862651 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:05:41.902102 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:41.902069 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 03 00:05:41.914509 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:41.914447 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0a1c13d-xcv5j"] Nov 03 00:05:41.926591 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:41.926549 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkp2v\" (UniqueName: \"kubernetes.io/projected/65ec560e-262c-4f97-836e-ccdaeb231102-kube-api-access-jkp2v\") pod \"maas-api-revoke-sub-a0a1c13d-xcv5j\" (UID: \"65ec560e-262c-4f97-836e-ccdaeb231102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a1c13d-xcv5j" Nov 03 00:05:41.926783 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:41.926626 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ec560e-262c-4f97-836e-ccdaeb231102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0a1c13d-xcv5j\" (UID: \"65ec560e-262c-4f97-836e-ccdaeb231102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a1c13d-xcv5j" Nov 03 00:05:41.926783 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:41.926654 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ec560e-262c-4f97-836e-ccdaeb231102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0a1c13d-xcv5j\" (UID: \"65ec560e-262c-4f97-836e-ccdaeb231102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a1c13d-xcv5j" Nov 03 00:05:41.927367 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:41.927336 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ec560e-262c-4f97-836e-ccdaeb231102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0a1c13d-xcv5j\" (UID: \"65ec560e-262c-4f97-836e-ccdaeb231102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a1c13d-xcv5j" Nov 03 00:05:41.929420 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:41.929398 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ec560e-262c-4f97-836e-ccdaeb231102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0a1c13d-xcv5j\" (UID: \"65ec560e-262c-4f97-836e-ccdaeb231102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a1c13d-xcv5j" Nov 03 00:05:41.987575 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:41.987537 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkp2v\" (UniqueName: \"kubernetes.io/projected/65ec560e-262c-4f97-836e-ccdaeb231102-kube-api-access-jkp2v\") pod \"maas-api-revoke-sub-a0a1c13d-xcv5j\" (UID: \"65ec560e-262c-4f97-836e-ccdaeb231102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a1c13d-xcv5j" Nov 03 00:05:42.123327 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:42.123210 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a1c13d-xcv5j" Nov 03 00:05:42.500915 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:42.500880 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0a1c13d-xcv5j"] Nov 03 00:05:42.503392 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:05:42.503359 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ec560e_262c_4f97_836e_ccdaeb231102.slice/crio-5c0db1889c2f5c4f098ede1a9e0b6fd2b65157ed9db8475f270765c465fcacc0 WatchSource:0}: Error finding container 5c0db1889c2f5c4f098ede1a9e0b6fd2b65157ed9db8475f270765c465fcacc0: Status 404 returned error can't find the container with id 5c0db1889c2f5c4f098ede1a9e0b6fd2b65157ed9db8475f270765c465fcacc0 Nov 03 00:05:42.752335 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:42.752222 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a1c13d-xcv5j" event={"ID":"65ec560e-262c-4f97-836e-ccdaeb231102","Type":"ContainerStarted","Data":"5c0db1889c2f5c4f098ede1a9e0b6fd2b65157ed9db8475f270765c465fcacc0"} Nov 03 00:05:43.156891 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:43.156847 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7ab7aff-x8lgm"] Nov 03 00:05:43.157365 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:43.157349 5114 cpu_manager.go:414] "containerMap: removing container" podUID="55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" containerName="istio-proxy" Nov 03 00:05:43.157406 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:43.157368 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" containerName="istio-proxy" Nov 03 00:05:43.157514 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:43.157502 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="55da13ab-fe6e-4a9c-a7a0-ecc82bfb4046" containerName="istio-proxy" Nov 03 00:05:43.159566 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:43.159547 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ab7aff-x8lgm" Nov 03 00:05:43.218633 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:43.218600 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7ab7aff-x8lgm"] Nov 03 00:05:43.241457 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:43.241406 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krblh\" (UniqueName: \"kubernetes.io/projected/741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6-kube-api-access-krblh\") pod \"maas-api-revoke-sub-c7ab7aff-x8lgm\" (UID: \"741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ab7aff-x8lgm" Nov 03 00:05:43.241694 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:43.241672 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7ab7aff-x8lgm\" (UID: \"741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ab7aff-x8lgm" Nov 03 00:05:43.241764 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:43.241741 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7ab7aff-x8lgm\" (UID: \"741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ab7aff-x8lgm" Nov 03 00:05:43.344317 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:43.343337 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krblh\" (UniqueName: \"kubernetes.io/projected/741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6-kube-api-access-krblh\") pod \"maas-api-revoke-sub-c7ab7aff-x8lgm\" (UID: \"741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ab7aff-x8lgm" Nov 03 00:05:43.344317 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:43.343545 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7ab7aff-x8lgm\" (UID: \"741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ab7aff-x8lgm" Nov 03 00:05:43.344592 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:43.344371 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7ab7aff-x8lgm\" (UID: \"741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ab7aff-x8lgm" Nov 03 00:05:43.345166 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:43.345141 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7ab7aff-x8lgm\" (UID: \"741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ab7aff-x8lgm" Nov 03 00:05:43.346795 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:43.346768 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7ab7aff-x8lgm\" (UID: \"741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ab7aff-x8lgm" Nov 03 00:05:43.405223 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:43.405191 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krblh\" (UniqueName: \"kubernetes.io/projected/741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6-kube-api-access-krblh\") pod \"maas-api-revoke-sub-c7ab7aff-x8lgm\" (UID: \"741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ab7aff-x8lgm" Nov 03 00:05:43.472841 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:43.472794 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ab7aff-x8lgm" Nov 03 00:05:43.759684 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:43.759589 5114 generic.go:356] "Generic (PLEG): container finished" podID="65ec560e-262c-4f97-836e-ccdaeb231102" containerID="8d6b000a29476736c4086bc524efcd7beac6114e46c34ad7242b595340e23143" exitCode=0 Nov 03 00:05:43.759876 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:43.759680 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a1c13d-xcv5j" event={"ID":"65ec560e-262c-4f97-836e-ccdaeb231102","Type":"ContainerDied","Data":"8d6b000a29476736c4086bc524efcd7beac6114e46c34ad7242b595340e23143"} Nov 03 00:05:43.848009 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:43.847985 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7ab7aff-x8lgm"] Nov 03 00:05:43.849993 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:05:43.849958 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741b89d4_3fa0_49e9_be5f_a5dcad5cc4c6.slice/crio-885766406a98aa7e67350a137bc490560c7a80e29c71a5301ef97660054788b7 WatchSource:0}: Error finding container 885766406a98aa7e67350a137bc490560c7a80e29c71a5301ef97660054788b7: Status 404 returned error can't find the container with id 885766406a98aa7e67350a137bc490560c7a80e29c71a5301ef97660054788b7 Nov 03 00:05:44.766867 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:44.766828 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ab7aff-x8lgm" event={"ID":"741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6","Type":"ContainerStarted","Data":"9abe1528fadb543f4d4138a34232b3d0f42d1f7d0bf33f61a1f55ade095778c3"} Nov 03 00:05:44.767297 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:44.766882 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ab7aff-x8lgm" event={"ID":"741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6","Type":"ContainerStarted","Data":"885766406a98aa7e67350a137bc490560c7a80e29c71a5301ef97660054788b7"} Nov 03 00:05:44.885522 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:44.885440 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ab7aff-x8lgm" podStartSLOduration=1.885420404 podStartE2EDuration="1.885420404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:44.882021088 +0000 UTC m=+4620.235054463" watchObservedRunningTime="2026-11-03 00:05:44.885420404 +0000 UTC m=+4620.238453786" Nov 03 00:05:44.932355 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:44.932326 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a1c13d-xcv5j" Nov 03 00:05:44.964698 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:44.964660 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ec560e-262c-4f97-836e-ccdaeb231102-kube-api-access-jkp2v\" (UniqueName: \"kubernetes.io/projected/65ec560e-262c-4f97-836e-ccdaeb231102-kube-api-access-jkp2v\") pod \"65ec560e-262c-4f97-836e-ccdaeb231102\" (UID: \"65ec560e-262c-4f97-836e-ccdaeb231102\") " Nov 03 00:05:44.964858 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:44.964783 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65ec560e-262c-4f97-836e-ccdaeb231102-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ec560e-262c-4f97-836e-ccdaeb231102-maas-api-xks-ca\") pod \"65ec560e-262c-4f97-836e-ccdaeb231102\" (UID: \"65ec560e-262c-4f97-836e-ccdaeb231102\") " Nov 03 00:05:44.964977 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:44.964945 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65ec560e-262c-4f97-836e-ccdaeb231102-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ec560e-262c-4f97-836e-ccdaeb231102-maas-api-openshift-ca\") pod \"65ec560e-262c-4f97-836e-ccdaeb231102\" (UID: \"65ec560e-262c-4f97-836e-ccdaeb231102\") " Nov 03 00:05:44.965332 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:44.965302 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ec560e-262c-4f97-836e-ccdaeb231102-maas-api-openshift-ca" pod "65ec560e-262c-4f97-836e-ccdaeb231102" (UID: "65ec560e-262c-4f97-836e-ccdaeb231102"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:05:44.965474 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:44.965450 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ec560e-262c-4f97-836e-ccdaeb231102-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:44.967064 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:44.967030 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ec560e-262c-4f97-836e-ccdaeb231102-kube-api-access-jkp2v" pod "65ec560e-262c-4f97-836e-ccdaeb231102" (UID: "65ec560e-262c-4f97-836e-ccdaeb231102"). InnerVolumeSpecName "kube-api-access-jkp2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:44.967189 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:44.967121 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65ec560e-262c-4f97-836e-ccdaeb231102-maas-api-xks-ca" pod "65ec560e-262c-4f97-836e-ccdaeb231102" (UID: "65ec560e-262c-4f97-836e-ccdaeb231102"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:05:45.066895 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:45.066852 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkp2v\" (UniqueName: \"kubernetes.io/projected/65ec560e-262c-4f97-836e-ccdaeb231102-kube-api-access-jkp2v\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:45.066895 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:45.066890 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ec560e-262c-4f97-836e-ccdaeb231102-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:45.774134 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:45.774097 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a1c13d-xcv5j" Nov 03 00:05:45.774134 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:45.774125 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a1c13d-xcv5j" event={"ID":"65ec560e-262c-4f97-836e-ccdaeb231102","Type":"ContainerDied","Data":"5c0db1889c2f5c4f098ede1a9e0b6fd2b65157ed9db8475f270765c465fcacc0"} Nov 03 00:05:45.774692 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:45.774205 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c0db1889c2f5c4f098ede1a9e0b6fd2b65157ed9db8475f270765c465fcacc0" Nov 03 00:05:45.775869 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:45.775842 5114 generic.go:356] "Generic (PLEG): container finished" podID="741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6" containerID="9abe1528fadb543f4d4138a34232b3d0f42d1f7d0bf33f61a1f55ade095778c3" exitCode=0 Nov 03 00:05:45.775996 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:45.775912 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ab7aff-x8lgm" event={"ID":"741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6","Type":"ContainerDied","Data":"9abe1528fadb543f4d4138a34232b3d0f42d1f7d0bf33f61a1f55ade095778c3"} Nov 03 00:05:46.766075 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:46.766040 5114 scope.go:122] "RemoveContainer" containerID="ae9b4e12c15b42b08b8f461f5f8f02a436d2213a5ffb46eee91822762b8f1b1a" Nov 03 00:05:46.776116 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:46.776082 5114 scope.go:122] "RemoveContainer" containerID="15eddbaa87f64d556ba0996b1af2693931305175f84627bc1157c04d0f3027c6" Nov 03 00:05:46.938913 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:46.938883 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ab7aff-x8lgm" Nov 03 00:05:46.987577 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:46.987538 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6-maas-api-openshift-ca\") pod \"741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6\" (UID: \"741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6\") " Nov 03 00:05:46.987773 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:46.987588 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6-maas-api-xks-ca\") pod \"741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6\" (UID: \"741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6\") " Nov 03 00:05:46.987773 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:46.987632 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6-kube-api-access-krblh\" (UniqueName: \"kubernetes.io/projected/741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6-kube-api-access-krblh\") pod \"741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6\" (UID: \"741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6\") " Nov 03 00:05:46.987972 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:46.987941 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6-maas-api-openshift-ca" pod "741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6" (UID: "741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:05:46.988196 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:46.988167 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:46.990069 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:46.990045 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6-kube-api-access-krblh" pod "741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6" (UID: "741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6"). InnerVolumeSpecName "kube-api-access-krblh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:46.990069 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:46.990047 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6-maas-api-xks-ca" pod "741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6" (UID: "741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:05:47.089115 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:47.089065 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:47.089115 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:47.089101 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krblh\" (UniqueName: \"kubernetes.io/projected/741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6-kube-api-access-krblh\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:47.789555 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:47.789527 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ab7aff-x8lgm" Nov 03 00:05:47.789555 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:47.789533 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ab7aff-x8lgm" event={"ID":"741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6","Type":"ContainerDied","Data":"885766406a98aa7e67350a137bc490560c7a80e29c71a5301ef97660054788b7"} Nov 03 00:05:47.790134 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:47.789578 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="885766406a98aa7e67350a137bc490560c7a80e29c71a5301ef97660054788b7" Nov 03 00:05:49.710347 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:49.710294 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" podUID="72301e09-ed03-4bc9-8264-028de7930a48" containerName="main" containerID="cri-o://b7ff04c47fefdab73a5e1a07e45869138da57d7832b4ee3c42eef1ed97b24e2e" gracePeriod=45 Nov 03 00:05:49.725057 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:49.725002 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" podUID="592a8e63-9584-4890-b38e-900b0482f735" containerName="main" containerID="cri-o://b737a12e0c60ff2393934b3cbd6557792ad48996348aafaaf51a96e12b36a616" gracePeriod=45 Nov 03 00:05:50.039609 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.039583 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 03 00:05:50.043001 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.042975 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 03 00:05:50.126926 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.126879 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-home\" (UniqueName: \"kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-home\") pod \"592a8e63-9584-4890-b38e-900b0482f735\" (UID: \"592a8e63-9584-4890-b38e-900b0482f735\") " Nov 03 00:05:50.127148 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.126965 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-dshm\" (UniqueName: \"kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-dshm\") pod \"72301e09-ed03-4bc9-8264-028de7930a48\" (UID: \"72301e09-ed03-4bc9-8264-028de7930a48\") " Nov 03 00:05:50.127148 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.127009 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-model-cache\") pod \"72301e09-ed03-4bc9-8264-028de7930a48\" (UID: \"72301e09-ed03-4bc9-8264-028de7930a48\") " Nov 03 00:05:50.127148 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.127072 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/592a8e63-9584-4890-b38e-900b0482f735-tls-certs\" (UniqueName: \"kubernetes.io/secret/592a8e63-9584-4890-b38e-900b0482f735-tls-certs\") pod \"592a8e63-9584-4890-b38e-900b0482f735\" (UID: \"592a8e63-9584-4890-b38e-900b0482f735\") " Nov 03 00:05:50.127148 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.127121 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72301e09-ed03-4bc9-8264-028de7930a48-tls-certs\" (UniqueName: \"kubernetes.io/secret/72301e09-ed03-4bc9-8264-028de7930a48-tls-certs\") pod \"72301e09-ed03-4bc9-8264-028de7930a48\" (UID: \"72301e09-ed03-4bc9-8264-028de7930a48\") " Nov 03 00:05:50.127375 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.127205 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-home" pod "592a8e63-9584-4890-b38e-900b0482f735" (UID: "592a8e63-9584-4890-b38e-900b0482f735"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:50.127375 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.127263 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-model-cache\") pod \"592a8e63-9584-4890-b38e-900b0482f735\" (UID: \"592a8e63-9584-4890-b38e-900b0482f735\") " Nov 03 00:05:50.127375 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.127316 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-model-cache" pod "72301e09-ed03-4bc9-8264-028de7930a48" (UID: "72301e09-ed03-4bc9-8264-028de7930a48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:50.127375 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.127330 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-home\" (UniqueName: \"kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-home\") pod \"72301e09-ed03-4bc9-8264-028de7930a48\" (UID: \"72301e09-ed03-4bc9-8264-028de7930a48\") " Nov 03 00:05:50.127619 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.127394 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-tmp-dir\") pod \"72301e09-ed03-4bc9-8264-028de7930a48\" (UID: \"72301e09-ed03-4bc9-8264-028de7930a48\") " Nov 03 00:05:50.127619 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.127463 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72301e09-ed03-4bc9-8264-028de7930a48-kube-api-access-wvcbk\" (UniqueName: \"kubernetes.io/projected/72301e09-ed03-4bc9-8264-028de7930a48-kube-api-access-wvcbk\") pod \"72301e09-ed03-4bc9-8264-028de7930a48\" (UID: \"72301e09-ed03-4bc9-8264-028de7930a48\") " Nov 03 00:05:50.127619 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.127515 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-model-cache" pod "592a8e63-9584-4890-b38e-900b0482f735" (UID: "592a8e63-9584-4890-b38e-900b0482f735"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:50.127619 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.127525 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/592a8e63-9584-4890-b38e-900b0482f735-kube-api-access-tgf7x\" (UniqueName: \"kubernetes.io/projected/592a8e63-9584-4890-b38e-900b0482f735-kube-api-access-tgf7x\") pod \"592a8e63-9584-4890-b38e-900b0482f735\" (UID: \"592a8e63-9584-4890-b38e-900b0482f735\") " Nov 03 00:05:50.127619 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.127578 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-tmp-dir\") pod \"592a8e63-9584-4890-b38e-900b0482f735\" (UID: \"592a8e63-9584-4890-b38e-900b0482f735\") " Nov 03 00:05:50.127911 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.127627 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-dshm\" (UniqueName: \"kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-dshm\") pod \"592a8e63-9584-4890-b38e-900b0482f735\" (UID: \"592a8e63-9584-4890-b38e-900b0482f735\") " Nov 03 00:05:50.128012 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.127578 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-home" pod "72301e09-ed03-4bc9-8264-028de7930a48" (UID: "72301e09-ed03-4bc9-8264-028de7930a48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:50.128134 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.127716 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-tmp-dir" pod "72301e09-ed03-4bc9-8264-028de7930a48" (UID: "72301e09-ed03-4bc9-8264-028de7930a48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:50.128209 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.127895 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-tmp-dir" pod "592a8e63-9584-4890-b38e-900b0482f735" (UID: "592a8e63-9584-4890-b38e-900b0482f735"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:50.128282 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.128260 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:50.128338 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.128290 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:50.128338 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.128308 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:50.128338 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.128325 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:50.128692 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.128338 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:50.128692 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.128352 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:50.130118 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.130089 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-dshm" pod "72301e09-ed03-4bc9-8264-028de7930a48" (UID: "72301e09-ed03-4bc9-8264-028de7930a48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:50.130301 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.130264 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72301e09-ed03-4bc9-8264-028de7930a48-tls-certs" pod "72301e09-ed03-4bc9-8264-028de7930a48" (UID: "72301e09-ed03-4bc9-8264-028de7930a48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:05:50.130395 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.130369 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/592a8e63-9584-4890-b38e-900b0482f735-tls-certs" pod "592a8e63-9584-4890-b38e-900b0482f735" (UID: "592a8e63-9584-4890-b38e-900b0482f735"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:05:50.130639 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.130615 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-dshm" pod "592a8e63-9584-4890-b38e-900b0482f735" (UID: "592a8e63-9584-4890-b38e-900b0482f735"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:50.130728 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.130654 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/592a8e63-9584-4890-b38e-900b0482f735-kube-api-access-tgf7x" pod "592a8e63-9584-4890-b38e-900b0482f735" (UID: "592a8e63-9584-4890-b38e-900b0482f735"). InnerVolumeSpecName "kube-api-access-tgf7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:50.130728 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.130685 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72301e09-ed03-4bc9-8264-028de7930a48-kube-api-access-wvcbk" pod "72301e09-ed03-4bc9-8264-028de7930a48" (UID: "72301e09-ed03-4bc9-8264-028de7930a48"). InnerVolumeSpecName "kube-api-access-wvcbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:50.229940 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.229903 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvcbk\" (UniqueName: \"kubernetes.io/projected/72301e09-ed03-4bc9-8264-028de7930a48-kube-api-access-wvcbk\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:50.229940 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.229933 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgf7x\" (UniqueName: \"kubernetes.io/projected/592a8e63-9584-4890-b38e-900b0482f735-kube-api-access-tgf7x\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:50.230150 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.229947 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/592a8e63-9584-4890-b38e-900b0482f735-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:50.230150 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.229961 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72301e09-ed03-4bc9-8264-028de7930a48-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:50.230150 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.229970 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/592a8e63-9584-4890-b38e-900b0482f735-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:50.230150 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.229978 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72301e09-ed03-4bc9-8264-028de7930a48-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:50.809676 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.809642 5114 generic.go:356] "Generic (PLEG): container finished" podID="72301e09-ed03-4bc9-8264-028de7930a48" containerID="b7ff04c47fefdab73a5e1a07e45869138da57d7832b4ee3c42eef1ed97b24e2e" exitCode=0 Nov 03 00:05:50.810190 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.809696 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" Nov 03 00:05:50.810190 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.809714 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" event={"ID":"72301e09-ed03-4bc9-8264-028de7930a48","Type":"ContainerDied","Data":"b7ff04c47fefdab73a5e1a07e45869138da57d7832b4ee3c42eef1ed97b24e2e"} Nov 03 00:05:50.810190 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.809758 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv" event={"ID":"72301e09-ed03-4bc9-8264-028de7930a48","Type":"ContainerDied","Data":"edde84b9657d9b6ba3b11dd16d346f50da2ab88a58c723efb0c0c7f3729cc273"} Nov 03 00:05:50.810190 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.809774 5114 scope.go:122] "RemoveContainer" containerID="b7ff04c47fefdab73a5e1a07e45869138da57d7832b4ee3c42eef1ed97b24e2e" Nov 03 00:05:50.811294 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.811274 5114 generic.go:356] "Generic (PLEG): container finished" podID="592a8e63-9584-4890-b38e-900b0482f735" containerID="b737a12e0c60ff2393934b3cbd6557792ad48996348aafaaf51a96e12b36a616" exitCode=0 Nov 03 00:05:50.811396 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.811314 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" Nov 03 00:05:50.811396 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.811364 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" event={"ID":"592a8e63-9584-4890-b38e-900b0482f735","Type":"ContainerDied","Data":"b737a12e0c60ff2393934b3cbd6557792ad48996348aafaaf51a96e12b36a616"} Nov 03 00:05:50.811524 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.811407 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt" event={"ID":"592a8e63-9584-4890-b38e-900b0482f735","Type":"ContainerDied","Data":"21c3c5d82fc7fa07a0de5053abd97b3c997273b2c3a9ea97ff829e80f98d4300"} Nov 03 00:05:50.819564 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.819539 5114 scope.go:122] "RemoveContainer" containerID="b7ff04c47fefdab73a5e1a07e45869138da57d7832b4ee3c42eef1ed97b24e2e" Nov 03 00:05:50.819915 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:05:50.819891 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7ff04c47fefdab73a5e1a07e45869138da57d7832b4ee3c42eef1ed97b24e2e\": container with ID starting with b7ff04c47fefdab73a5e1a07e45869138da57d7832b4ee3c42eef1ed97b24e2e not found: ID does not exist" containerID="b7ff04c47fefdab73a5e1a07e45869138da57d7832b4ee3c42eef1ed97b24e2e" Nov 03 00:05:50.819966 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.819929 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7ff04c47fefdab73a5e1a07e45869138da57d7832b4ee3c42eef1ed97b24e2e"} err="failed to get container status \"b7ff04c47fefdab73a5e1a07e45869138da57d7832b4ee3c42eef1ed97b24e2e\": rpc error: code = NotFound desc = could not find container \"b7ff04c47fefdab73a5e1a07e45869138da57d7832b4ee3c42eef1ed97b24e2e\": container with ID starting with b7ff04c47fefdab73a5e1a07e45869138da57d7832b4ee3c42eef1ed97b24e2e not found: ID does not exist" Nov 03 00:05:50.819966 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.819947 5114 scope.go:122] "RemoveContainer" containerID="b737a12e0c60ff2393934b3cbd6557792ad48996348aafaaf51a96e12b36a616" Nov 03 00:05:50.829761 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.829738 5114 scope.go:122] "RemoveContainer" containerID="b737a12e0c60ff2393934b3cbd6557792ad48996348aafaaf51a96e12b36a616" Nov 03 00:05:50.830108 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:05:50.830084 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b737a12e0c60ff2393934b3cbd6557792ad48996348aafaaf51a96e12b36a616\": container with ID starting with b737a12e0c60ff2393934b3cbd6557792ad48996348aafaaf51a96e12b36a616 not found: ID does not exist" containerID="b737a12e0c60ff2393934b3cbd6557792ad48996348aafaaf51a96e12b36a616" Nov 03 00:05:50.830185 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.830116 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b737a12e0c60ff2393934b3cbd6557792ad48996348aafaaf51a96e12b36a616"} err="failed to get container status \"b737a12e0c60ff2393934b3cbd6557792ad48996348aafaaf51a96e12b36a616\": rpc error: code = NotFound desc = could not find container \"b737a12e0c60ff2393934b3cbd6557792ad48996348aafaaf51a96e12b36a616\": container with ID starting with b737a12e0c60ff2393934b3cbd6557792ad48996348aafaaf51a96e12b36a616 not found: ID does not exist" Nov 03 00:05:50.987724 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.987686 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv"] Nov 03 00:05:50.994232 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:50.994199 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ec2f9a/premium-w0-ec2f9a-kserve-f9dd6557b-nmbxv"] Nov 03 00:05:51.105399 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:51.105294 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt"] Nov 03 00:05:51.107718 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:51.107678 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ec2f9a/simulator-w0-ec2f9a-kserve-5dfbbd494f-xfbjt"] Nov 03 00:05:52.793224 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:52.793182 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="592a8e63-9584-4890-b38e-900b0482f735" path="/var/lib/kubelet/pods/592a8e63-9584-4890-b38e-900b0482f735/volumes" Nov 03 00:05:52.793787 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:52.793767 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72301e09-ed03-4bc9-8264-028de7930a48" path="/var/lib/kubelet/pods/72301e09-ed03-4bc9-8264-028de7930a48/volumes" Nov 03 00:05:53.881810 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:53.881772 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l"] Nov 03 00:05:55.409413 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:55.409371 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7"] Nov 03 00:05:58.047741 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:58.047701 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ed38add-m6dtb"] Nov 03 00:05:58.096886 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:58.096849 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ed38add-m6dtb"] Nov 03 00:05:58.793290 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:05:58.793256 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0e0537f-9d0d-4179-8329-85218b5c9949" path="/var/lib/kubelet/pods/b0e0537f-9d0d-4179-8329-85218b5c9949/volumes" Nov 03 00:06:07.585209 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:06:07.585164 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:06:08.906053 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:08.906000 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" podUID="6db69f57-95f7-4970-a795-0b31d9be79ed" containerName="main" containerID="cri-o://fbeac8c5bc4956307952b52fe3e8fd63e467f245fb123c458cdf439e63e102b3" gracePeriod=45 Nov 03 00:06:09.177946 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.177916 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:06:09.356520 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.356449 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-tmp-dir\") pod \"6db69f57-95f7-4970-a795-0b31d9be79ed\" (UID: \"6db69f57-95f7-4970-a795-0b31d9be79ed\") " Nov 03 00:06:09.356739 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.356569 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-dshm\") pod \"6db69f57-95f7-4970-a795-0b31d9be79ed\" (UID: \"6db69f57-95f7-4970-a795-0b31d9be79ed\") " Nov 03 00:06:09.356739 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.356639 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-model-cache\") pod \"6db69f57-95f7-4970-a795-0b31d9be79ed\" (UID: \"6db69f57-95f7-4970-a795-0b31d9be79ed\") " Nov 03 00:06:09.356739 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.356709 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-home\" (UniqueName: \"kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-home\") pod \"6db69f57-95f7-4970-a795-0b31d9be79ed\" (UID: \"6db69f57-95f7-4970-a795-0b31d9be79ed\") " Nov 03 00:06:09.356919 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.356770 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db69f57-95f7-4970-a795-0b31d9be79ed-kube-api-access-hvdfw\" (UniqueName: \"kubernetes.io/projected/6db69f57-95f7-4970-a795-0b31d9be79ed-kube-api-access-hvdfw\") pod \"6db69f57-95f7-4970-a795-0b31d9be79ed\" (UID: \"6db69f57-95f7-4970-a795-0b31d9be79ed\") " Nov 03 00:06:09.356919 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.356770 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-tmp-dir" pod "6db69f57-95f7-4970-a795-0b31d9be79ed" (UID: "6db69f57-95f7-4970-a795-0b31d9be79ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:09.356919 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.356874 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6db69f57-95f7-4970-a795-0b31d9be79ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/6db69f57-95f7-4970-a795-0b31d9be79ed-tls-certs\") pod \"6db69f57-95f7-4970-a795-0b31d9be79ed\" (UID: \"6db69f57-95f7-4970-a795-0b31d9be79ed\") " Nov 03 00:06:09.357094 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.356972 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-home" pod "6db69f57-95f7-4970-a795-0b31d9be79ed" (UID: "6db69f57-95f7-4970-a795-0b31d9be79ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:09.357094 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.356979 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-model-cache" pod "6db69f57-95f7-4970-a795-0b31d9be79ed" (UID: "6db69f57-95f7-4970-a795-0b31d9be79ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:09.357299 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.357281 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:09.357370 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.357304 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:09.357370 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.357318 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:09.358994 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.358952 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db69f57-95f7-4970-a795-0b31d9be79ed-kube-api-access-hvdfw" pod "6db69f57-95f7-4970-a795-0b31d9be79ed" (UID: "6db69f57-95f7-4970-a795-0b31d9be79ed"). InnerVolumeSpecName "kube-api-access-hvdfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:09.359128 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.358989 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-dshm" pod "6db69f57-95f7-4970-a795-0b31d9be79ed" (UID: "6db69f57-95f7-4970-a795-0b31d9be79ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:09.359128 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.359039 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db69f57-95f7-4970-a795-0b31d9be79ed-tls-certs" pod "6db69f57-95f7-4970-a795-0b31d9be79ed" (UID: "6db69f57-95f7-4970-a795-0b31d9be79ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:09.458668 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.458576 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db69f57-95f7-4970-a795-0b31d9be79ed-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:09.458668 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.458612 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvdfw\" (UniqueName: \"kubernetes.io/projected/6db69f57-95f7-4970-a795-0b31d9be79ed-kube-api-access-hvdfw\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:09.458668 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.458627 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db69f57-95f7-4970-a795-0b31d9be79ed-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:09.925752 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.925712 5114 generic.go:356] "Generic (PLEG): container finished" podID="6db69f57-95f7-4970-a795-0b31d9be79ed" containerID="fbeac8c5bc4956307952b52fe3e8fd63e467f245fb123c458cdf439e63e102b3" exitCode=0 Nov 03 00:06:09.926215 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.925759 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" Nov 03 00:06:09.926215 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.925798 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" event={"ID":"6db69f57-95f7-4970-a795-0b31d9be79ed","Type":"ContainerDied","Data":"fbeac8c5bc4956307952b52fe3e8fd63e467f245fb123c458cdf439e63e102b3"} Nov 03 00:06:09.926215 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.925844 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" event={"ID":"6db69f57-95f7-4970-a795-0b31d9be79ed","Type":"ContainerDied","Data":"60632f956e29b128066937155f03784f5582ca78577a44eeee1f8ec189d093e6"} Nov 03 00:06:09.926215 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.925864 5114 scope.go:122] "RemoveContainer" containerID="fbeac8c5bc4956307952b52fe3e8fd63e467f245fb123c458cdf439e63e102b3" Nov 03 00:06:09.936705 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.936680 5114 scope.go:122] "RemoveContainer" containerID="fbeac8c5bc4956307952b52fe3e8fd63e467f245fb123c458cdf439e63e102b3" Nov 03 00:06:09.937081 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:06:09.937058 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbeac8c5bc4956307952b52fe3e8fd63e467f245fb123c458cdf439e63e102b3\": container with ID starting with fbeac8c5bc4956307952b52fe3e8fd63e467f245fb123c458cdf439e63e102b3 not found: ID does not exist" containerID="fbeac8c5bc4956307952b52fe3e8fd63e467f245fb123c458cdf439e63e102b3" Nov 03 00:06:09.937121 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:09.937093 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbeac8c5bc4956307952b52fe3e8fd63e467f245fb123c458cdf439e63e102b3"} err="failed to get container status \"fbeac8c5bc4956307952b52fe3e8fd63e467f245fb123c458cdf439e63e102b3\": rpc error: code = NotFound desc = could not find container \"fbeac8c5bc4956307952b52fe3e8fd63e467f245fb123c458cdf439e63e102b3\": container with ID starting with fbeac8c5bc4956307952b52fe3e8fd63e467f245fb123c458cdf439e63e102b3 not found: ID does not exist" Nov 03 00:06:10.088726 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.088683 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l"] Nov 03 00:06:10.092362 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.092321 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l"] Nov 03 00:06:10.121789 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.121743 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-980b82/rate-test-model-980b82-kserve-57679ddf77-pkc7l" podUID="6db69f57-95f7-4970-a795-0b31d9be79ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.172:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:06:10.436723 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.436678 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" podUID="439f5c77-44d7-46b9-8b77-daa3df61b471" containerName="main" containerID="cri-o://b64495ed8beb13a3033b4eca61e22789d699a6723c282a378b81ce1b50507f24" gracePeriod=45 Nov 03 00:06:10.536300 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.536257 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" podUID="439f5c77-44d7-46b9-8b77-daa3df61b471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.173:8000/ready\": dial tcp 10.134.0.173:8000: connect: connection refused" Nov 03 00:06:10.701586 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.701500 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:06:10.771173 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.771132 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/439f5c77-44d7-46b9-8b77-daa3df61b471-tls-certs\" (UniqueName: \"kubernetes.io/secret/439f5c77-44d7-46b9-8b77-daa3df61b471-tls-certs\") pod \"439f5c77-44d7-46b9-8b77-daa3df61b471\" (UID: \"439f5c77-44d7-46b9-8b77-daa3df61b471\") " Nov 03 00:06:10.771362 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.771253 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-tmp-dir\") pod \"439f5c77-44d7-46b9-8b77-daa3df61b471\" (UID: \"439f5c77-44d7-46b9-8b77-daa3df61b471\") " Nov 03 00:06:10.771362 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.771286 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-dshm\" (UniqueName: \"kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-dshm\") pod \"439f5c77-44d7-46b9-8b77-daa3df61b471\" (UID: \"439f5c77-44d7-46b9-8b77-daa3df61b471\") " Nov 03 00:06:10.771449 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.771364 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/439f5c77-44d7-46b9-8b77-daa3df61b471-kube-api-access-xvns4\" (UniqueName: \"kubernetes.io/projected/439f5c77-44d7-46b9-8b77-daa3df61b471-kube-api-access-xvns4\") pod \"439f5c77-44d7-46b9-8b77-daa3df61b471\" (UID: \"439f5c77-44d7-46b9-8b77-daa3df61b471\") " Nov 03 00:06:10.771449 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.771399 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-model-cache\") pod \"439f5c77-44d7-46b9-8b77-daa3df61b471\" (UID: \"439f5c77-44d7-46b9-8b77-daa3df61b471\") " Nov 03 00:06:10.771583 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.771544 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-home\" (UniqueName: \"kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-home\") pod \"439f5c77-44d7-46b9-8b77-daa3df61b471\" (UID: \"439f5c77-44d7-46b9-8b77-daa3df61b471\") " Nov 03 00:06:10.771645 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.771611 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-tmp-dir" pod "439f5c77-44d7-46b9-8b77-daa3df61b471" (UID: "439f5c77-44d7-46b9-8b77-daa3df61b471"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:10.771711 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.771692 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-model-cache" pod "439f5c77-44d7-46b9-8b77-daa3df61b471" (UID: "439f5c77-44d7-46b9-8b77-daa3df61b471"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:10.771858 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.771828 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-home" pod "439f5c77-44d7-46b9-8b77-daa3df61b471" (UID: "439f5c77-44d7-46b9-8b77-daa3df61b471"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:10.772010 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.771982 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:10.772010 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.772001 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:10.772303 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.772014 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:10.773717 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.773690 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/439f5c77-44d7-46b9-8b77-daa3df61b471-tls-certs" pod "439f5c77-44d7-46b9-8b77-daa3df61b471" (UID: "439f5c77-44d7-46b9-8b77-daa3df61b471"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:10.773828 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.773743 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-dshm" pod "439f5c77-44d7-46b9-8b77-daa3df61b471" (UID: "439f5c77-44d7-46b9-8b77-daa3df61b471"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:10.773877 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.773839 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439f5c77-44d7-46b9-8b77-daa3df61b471-kube-api-access-xvns4" pod "439f5c77-44d7-46b9-8b77-daa3df61b471" (UID: "439f5c77-44d7-46b9-8b77-daa3df61b471"). InnerVolumeSpecName "kube-api-access-xvns4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:10.793745 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.793710 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db69f57-95f7-4970-a795-0b31d9be79ed" path="/var/lib/kubelet/pods/6db69f57-95f7-4970-a795-0b31d9be79ed/volumes" Nov 03 00:06:10.872984 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.872926 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/439f5c77-44d7-46b9-8b77-daa3df61b471-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:10.872984 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.872975 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvns4\" (UniqueName: \"kubernetes.io/projected/439f5c77-44d7-46b9-8b77-daa3df61b471-kube-api-access-xvns4\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:10.872984 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.872990 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439f5c77-44d7-46b9-8b77-daa3df61b471-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:10.933617 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.933576 5114 generic.go:356] "Generic (PLEG): container finished" podID="439f5c77-44d7-46b9-8b77-daa3df61b471" containerID="b64495ed8beb13a3033b4eca61e22789d699a6723c282a378b81ce1b50507f24" exitCode=0 Nov 03 00:06:10.934107 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.933630 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" Nov 03 00:06:10.934107 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.933667 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" event={"ID":"439f5c77-44d7-46b9-8b77-daa3df61b471","Type":"ContainerDied","Data":"b64495ed8beb13a3033b4eca61e22789d699a6723c282a378b81ce1b50507f24"} Nov 03 00:06:10.934107 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.933713 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7" event={"ID":"439f5c77-44d7-46b9-8b77-daa3df61b471","Type":"ContainerDied","Data":"3341d2f98a250c438952107cf4bdc6b364fa25b92730297ded425e0b15e4a534"} Nov 03 00:06:10.934107 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.933736 5114 scope.go:122] "RemoveContainer" containerID="b64495ed8beb13a3033b4eca61e22789d699a6723c282a378b81ce1b50507f24" Nov 03 00:06:10.943330 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.943306 5114 scope.go:122] "RemoveContainer" containerID="b64495ed8beb13a3033b4eca61e22789d699a6723c282a378b81ce1b50507f24" Nov 03 00:06:10.943710 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:06:10.943665 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b64495ed8beb13a3033b4eca61e22789d699a6723c282a378b81ce1b50507f24\": container with ID starting with b64495ed8beb13a3033b4eca61e22789d699a6723c282a378b81ce1b50507f24 not found: ID does not exist" containerID="b64495ed8beb13a3033b4eca61e22789d699a6723c282a378b81ce1b50507f24" Nov 03 00:06:10.943848 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:10.943710 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b64495ed8beb13a3033b4eca61e22789d699a6723c282a378b81ce1b50507f24"} err="failed to get container status \"b64495ed8beb13a3033b4eca61e22789d699a6723c282a378b81ce1b50507f24\": rpc error: code = NotFound desc = could not find container \"b64495ed8beb13a3033b4eca61e22789d699a6723c282a378b81ce1b50507f24\": container with ID starting with b64495ed8beb13a3033b4eca61e22789d699a6723c282a378b81ce1b50507f24 not found: ID does not exist" Nov 03 00:06:11.097911 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:11.097867 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7"] Nov 03 00:06:11.101876 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:11.101844 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8d3cb/rate-test-model-b8d3cb-kserve-745cd6d89c-hrpn7"] Nov 03 00:06:12.805005 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:12.804957 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="439f5c77-44d7-46b9-8b77-daa3df61b471" path="/var/lib/kubelet/pods/439f5c77-44d7-46b9-8b77-daa3df61b471/volumes" Nov 03 00:06:26.330860 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.330821 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw"] Nov 03 00:06:26.331397 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.331360 5114 cpu_manager.go:414] "containerMap: removing container" podUID="439f5c77-44d7-46b9-8b77-daa3df61b471" containerName="main" Nov 03 00:06:26.331397 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.331384 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="439f5c77-44d7-46b9-8b77-daa3df61b471" containerName="main" Nov 03 00:06:26.331516 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.331408 5114 cpu_manager.go:414] "containerMap: removing container" podUID="72301e09-ed03-4bc9-8264-028de7930a48" containerName="main" Nov 03 00:06:26.331516 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.331413 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72301e09-ed03-4bc9-8264-028de7930a48" containerName="main" Nov 03 00:06:26.331516 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.331428 5114 cpu_manager.go:414] "containerMap: removing container" podUID="6db69f57-95f7-4970-a795-0b31d9be79ed" containerName="main" Nov 03 00:06:26.331516 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.331432 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db69f57-95f7-4970-a795-0b31d9be79ed" containerName="main" Nov 03 00:06:26.331516 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.331451 5114 cpu_manager.go:414] "containerMap: removing container" podUID="592a8e63-9584-4890-b38e-900b0482f735" containerName="main" Nov 03 00:06:26.331516 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.331457 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="592a8e63-9584-4890-b38e-900b0482f735" containerName="main" Nov 03 00:06:26.331516 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.331465 5114 cpu_manager.go:414] "containerMap: removing container" podUID="65ec560e-262c-4f97-836e-ccdaeb231102" containerName="revoke-keys" Nov 03 00:06:26.331516 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.331469 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ec560e-262c-4f97-836e-ccdaeb231102" containerName="revoke-keys" Nov 03 00:06:26.331516 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.331508 5114 cpu_manager.go:414] "containerMap: removing container" podUID="741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6" containerName="revoke-keys" Nov 03 00:06:26.331516 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.331513 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6" containerName="revoke-keys" Nov 03 00:06:26.331983 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.331592 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6" containerName="revoke-keys" Nov 03 00:06:26.331983 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.331603 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="439f5c77-44d7-46b9-8b77-daa3df61b471" containerName="main" Nov 03 00:06:26.331983 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.331617 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="72301e09-ed03-4bc9-8264-028de7930a48" containerName="main" Nov 03 00:06:26.331983 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.331624 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db69f57-95f7-4970-a795-0b31d9be79ed" containerName="main" Nov 03 00:06:26.331983 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.331631 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="592a8e63-9584-4890-b38e-900b0482f735" containerName="main" Nov 03 00:06:26.331983 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.331637 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ec560e-262c-4f97-836e-ccdaeb231102" containerName="revoke-keys" Nov 03 00:06:26.334839 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.334815 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.388178 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.388140 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ec2f9a-openshift-default-dockercfg-4nt5j\"" Nov 03 00:06:26.441073 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.441030 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw"] Nov 03 00:06:26.458425 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.458382 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.458626 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.458449 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a85aec-06e9-4952-9a16-055faf8cf19b-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.458626 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.458603 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmjkm\" (UniqueName: \"kubernetes.io/projected/65a85aec-06e9-4952-9a16-055faf8cf19b-kube-api-access-vmjkm\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.458794 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.458636 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.458856 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.458835 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.458927 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.458872 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.458999 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.458958 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.459046 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.459023 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.459085 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.459060 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.560796 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.560744 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a85aec-06e9-4952-9a16-055faf8cf19b-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.561063 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.560813 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmjkm\" (UniqueName: \"kubernetes.io/projected/65a85aec-06e9-4952-9a16-055faf8cf19b-kube-api-access-vmjkm\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.561063 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.560843 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.561063 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.560977 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.561063 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.561002 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.561063 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.561040 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.561423 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.561068 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.561423 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.561100 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.561423 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.561150 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.561604 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.561470 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.561647 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.561603 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.561647 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.561624 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.561783 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.561706 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a85aec-06e9-4952-9a16-055faf8cf19b-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.562056 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.562030 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.563914 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.563889 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.564133 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.564109 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.622188 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.622114 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmjkm\" (UniqueName: \"kubernetes.io/projected/65a85aec-06e9-4952-9a16-055faf8cf19b-kube-api-access-vmjkm\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.622370 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.622258 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:26.652018 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:26.651978 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:27.085497 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:27.085459 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw"] Nov 03 00:06:27.087280 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:06:27.087251 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a85aec_06e9_4952_9a16_055faf8cf19b.slice/crio-574480419f5959b194aa57e598a59fdfc16ebe63d3c5cc8a3d35e4ce4ba83f6f WatchSource:0}: Error finding container 574480419f5959b194aa57e598a59fdfc16ebe63d3c5cc8a3d35e4ce4ba83f6f: Status 404 returned error can't find the container with id 574480419f5959b194aa57e598a59fdfc16ebe63d3c5cc8a3d35e4ce4ba83f6f Nov 03 00:06:27.089956 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:27.089917 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:06:27.090080 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:27.090008 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:06:27.090080 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:27.090058 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:06:28.052326 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:28.052286 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" event={"ID":"65a85aec-06e9-4952-9a16-055faf8cf19b","Type":"ContainerStarted","Data":"0152612b60723f99b5b0489177ddb16ef9125db6c9d9138d8d1e082943bd2dc6"} Nov 03 00:06:28.052873 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:28.052370 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" event={"ID":"65a85aec-06e9-4952-9a16-055faf8cf19b","Type":"ContainerStarted","Data":"574480419f5959b194aa57e598a59fdfc16ebe63d3c5cc8a3d35e4ce4ba83f6f"} Nov 03 00:06:28.275368 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:28.275302 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" podStartSLOduration=2.275285713 podStartE2EDuration="2.275285713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:06:28.223982578 +0000 UTC m=+4663.577015954" watchObservedRunningTime="2026-11-03 00:06:28.275285713 +0000 UTC m=+4663.628319085" Nov 03 00:06:28.652918 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:28.652876 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:28.654948 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:28.654915 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 00:06:28.655117 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:28.654975 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" podUID="65a85aec-06e9-4952-9a16-055faf8cf19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 00:06:29.652968 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:29.652922 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 00:06:29.653416 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:29.652994 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" podUID="65a85aec-06e9-4952-9a16-055faf8cf19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 00:06:30.653340 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:30.653298 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 00:06:30.653885 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:30.653369 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" podUID="65a85aec-06e9-4952-9a16-055faf8cf19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 00:06:31.653312 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:31.653271 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 00:06:31.653573 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:31.653336 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" podUID="65a85aec-06e9-4952-9a16-055faf8cf19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 00:06:32.653475 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:32.653426 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 00:06:32.653695 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:32.653517 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" podUID="65a85aec-06e9-4952-9a16-055faf8cf19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 00:06:33.653102 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:33.653052 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 00:06:33.653305 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:33.653132 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" podUID="65a85aec-06e9-4952-9a16-055faf8cf19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 00:06:34.658353 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:34.658311 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:34.659571 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:34.659316 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:34.660220 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:34.660198 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:36.272056 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:36.272018 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9b63b44-wlvss"] Nov 03 00:06:36.275791 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:36.275772 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b63b44-wlvss" Nov 03 00:06:36.305183 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:36.305142 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc2nv\" (UniqueName: \"kubernetes.io/projected/ab305d6b-09c6-4e53-94fa-49e42facf5f4-kube-api-access-cc2nv\") pod \"maas-api-revoke-sub-a9b63b44-wlvss\" (UID: \"ab305d6b-09c6-4e53-94fa-49e42facf5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b63b44-wlvss" Nov 03 00:06:36.305407 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:36.305194 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab305d6b-09c6-4e53-94fa-49e42facf5f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9b63b44-wlvss\" (UID: \"ab305d6b-09c6-4e53-94fa-49e42facf5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b63b44-wlvss" Nov 03 00:06:36.305407 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:36.305279 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab305d6b-09c6-4e53-94fa-49e42facf5f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9b63b44-wlvss\" (UID: \"ab305d6b-09c6-4e53-94fa-49e42facf5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b63b44-wlvss" Nov 03 00:06:36.330216 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:36.330171 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:06:36.379917 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:36.379875 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 03 00:06:36.382288 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:36.382260 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9b63b44-wlvss"] Nov 03 00:06:36.406904 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:36.406858 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc2nv\" (UniqueName: \"kubernetes.io/projected/ab305d6b-09c6-4e53-94fa-49e42facf5f4-kube-api-access-cc2nv\") pod \"maas-api-revoke-sub-a9b63b44-wlvss\" (UID: \"ab305d6b-09c6-4e53-94fa-49e42facf5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b63b44-wlvss" Nov 03 00:06:36.407163 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:36.406909 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab305d6b-09c6-4e53-94fa-49e42facf5f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9b63b44-wlvss\" (UID: \"ab305d6b-09c6-4e53-94fa-49e42facf5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b63b44-wlvss" Nov 03 00:06:36.407163 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:36.406945 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab305d6b-09c6-4e53-94fa-49e42facf5f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9b63b44-wlvss\" (UID: \"ab305d6b-09c6-4e53-94fa-49e42facf5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b63b44-wlvss" Nov 03 00:06:36.407744 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:36.407717 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab305d6b-09c6-4e53-94fa-49e42facf5f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9b63b44-wlvss\" (UID: \"ab305d6b-09c6-4e53-94fa-49e42facf5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b63b44-wlvss" Nov 03 00:06:36.409851 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:36.409809 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab305d6b-09c6-4e53-94fa-49e42facf5f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9b63b44-wlvss\" (UID: \"ab305d6b-09c6-4e53-94fa-49e42facf5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b63b44-wlvss" Nov 03 00:06:36.468110 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:36.468081 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc2nv\" (UniqueName: \"kubernetes.io/projected/ab305d6b-09c6-4e53-94fa-49e42facf5f4-kube-api-access-cc2nv\") pod \"maas-api-revoke-sub-a9b63b44-wlvss\" (UID: \"ab305d6b-09c6-4e53-94fa-49e42facf5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b63b44-wlvss" Nov 03 00:06:36.590667 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:36.590626 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b63b44-wlvss" Nov 03 00:06:36.970614 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:36.970586 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9b63b44-wlvss"] Nov 03 00:06:36.972731 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:06:36.972694 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab305d6b_09c6_4e53_94fa_49e42facf5f4.slice/crio-81fdb9adca74922000f1fb1b8aa4fe680b3d3991716698a14ebba10612d1bd9a WatchSource:0}: Error finding container 81fdb9adca74922000f1fb1b8aa4fe680b3d3991716698a14ebba10612d1bd9a: Status 404 returned error can't find the container with id 81fdb9adca74922000f1fb1b8aa4fe680b3d3991716698a14ebba10612d1bd9a Nov 03 00:06:37.116509 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:37.116451 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b63b44-wlvss" event={"ID":"ab305d6b-09c6-4e53-94fa-49e42facf5f4","Type":"ContainerStarted","Data":"81fdb9adca74922000f1fb1b8aa4fe680b3d3991716698a14ebba10612d1bd9a"} Nov 03 00:06:37.584579 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:06:37.584534 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:06:38.124327 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:38.124292 5114 generic.go:356] "Generic (PLEG): container finished" podID="ab305d6b-09c6-4e53-94fa-49e42facf5f4" containerID="d5678b2acc5340e863ce856a37b59ea6bd6e7b93a49cd473c080e927b25a89d5" exitCode=0 Nov 03 00:06:38.124538 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:38.124377 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b63b44-wlvss" event={"ID":"ab305d6b-09c6-4e53-94fa-49e42facf5f4","Type":"ContainerDied","Data":"d5678b2acc5340e863ce856a37b59ea6bd6e7b93a49cd473c080e927b25a89d5"} Nov 03 00:06:39.273313 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:39.273282 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b63b44-wlvss" Nov 03 00:06:39.335332 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:39.335284 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab305d6b-09c6-4e53-94fa-49e42facf5f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab305d6b-09c6-4e53-94fa-49e42facf5f4-maas-api-openshift-ca\") pod \"ab305d6b-09c6-4e53-94fa-49e42facf5f4\" (UID: \"ab305d6b-09c6-4e53-94fa-49e42facf5f4\") " Nov 03 00:06:39.335561 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:39.335507 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab305d6b-09c6-4e53-94fa-49e42facf5f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab305d6b-09c6-4e53-94fa-49e42facf5f4-maas-api-xks-ca\") pod \"ab305d6b-09c6-4e53-94fa-49e42facf5f4\" (UID: \"ab305d6b-09c6-4e53-94fa-49e42facf5f4\") " Nov 03 00:06:39.335645 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:39.335581 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab305d6b-09c6-4e53-94fa-49e42facf5f4-kube-api-access-cc2nv\" (UniqueName: \"kubernetes.io/projected/ab305d6b-09c6-4e53-94fa-49e42facf5f4-kube-api-access-cc2nv\") pod \"ab305d6b-09c6-4e53-94fa-49e42facf5f4\" (UID: \"ab305d6b-09c6-4e53-94fa-49e42facf5f4\") " Nov 03 00:06:39.336007 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:39.335981 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab305d6b-09c6-4e53-94fa-49e42facf5f4-maas-api-openshift-ca" pod "ab305d6b-09c6-4e53-94fa-49e42facf5f4" (UID: "ab305d6b-09c6-4e53-94fa-49e42facf5f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:39.337968 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:39.337939 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab305d6b-09c6-4e53-94fa-49e42facf5f4-maas-api-xks-ca" pod "ab305d6b-09c6-4e53-94fa-49e42facf5f4" (UID: "ab305d6b-09c6-4e53-94fa-49e42facf5f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:39.338068 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:39.337979 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab305d6b-09c6-4e53-94fa-49e42facf5f4-kube-api-access-cc2nv" pod "ab305d6b-09c6-4e53-94fa-49e42facf5f4" (UID: "ab305d6b-09c6-4e53-94fa-49e42facf5f4"). InnerVolumeSpecName "kube-api-access-cc2nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:39.436612 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:39.436516 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc2nv\" (UniqueName: \"kubernetes.io/projected/ab305d6b-09c6-4e53-94fa-49e42facf5f4-kube-api-access-cc2nv\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:39.436612 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:39.436550 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab305d6b-09c6-4e53-94fa-49e42facf5f4-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:39.436612 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:39.436562 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab305d6b-09c6-4e53-94fa-49e42facf5f4-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:40.137064 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:40.137033 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b63b44-wlvss" Nov 03 00:06:40.137275 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:40.137065 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b63b44-wlvss" event={"ID":"ab305d6b-09c6-4e53-94fa-49e42facf5f4","Type":"ContainerDied","Data":"81fdb9adca74922000f1fb1b8aa4fe680b3d3991716698a14ebba10612d1bd9a"} Nov 03 00:06:40.137275 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:40.137126 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81fdb9adca74922000f1fb1b8aa4fe680b3d3991716698a14ebba10612d1bd9a" Nov 03 00:06:40.603857 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:40.603819 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd"] Nov 03 00:06:40.621742 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:40.621698 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t"] Nov 03 00:06:46.813009 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:46.812977 5114 scope.go:122] "RemoveContainer" containerID="dee311b64354a4dfbbdec9b7b352dc86ba0a98b705ab864e2caca6e27b3326b9" Nov 03 00:06:48.981592 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:48.981555 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw"] Nov 03 00:06:48.982107 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:48.981773 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" podUID="65a85aec-06e9-4952-9a16-055faf8cf19b" containerName="istio-proxy" containerID="cri-o://0152612b60723f99b5b0489177ddb16ef9125db6c9d9138d8d1e082943bd2dc6" gracePeriod=30 Nov 03 00:06:49.464728 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.464689 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn"] Nov 03 00:06:49.465141 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.465127 5114 cpu_manager.go:414] "containerMap: removing container" podUID="ab305d6b-09c6-4e53-94fa-49e42facf5f4" containerName="revoke-keys" Nov 03 00:06:49.465194 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.465142 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab305d6b-09c6-4e53-94fa-49e42facf5f4" containerName="revoke-keys" Nov 03 00:06:49.465250 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.465241 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab305d6b-09c6-4e53-94fa-49e42facf5f4" containerName="revoke-keys" Nov 03 00:06:49.469793 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.469771 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:06:49.525060 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.525027 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-980b82\"/\"openshift-service-ca.crt\"" Nov 03 00:06:49.525302 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.525027 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-980b82\"/\"kube-root-ca.crt\"" Nov 03 00:06:49.527322 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.527294 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn"] Nov 03 00:06:49.557684 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.557646 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-tmp-dir\") pod \"sub-test-model-980b82-kserve-5968685f87-r92jn\" (UID: \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\") " pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:06:49.557876 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.557703 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-model-cache\") pod \"sub-test-model-980b82-kserve-5968685f87-r92jn\" (UID: \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\") " pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:06:49.557924 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.557892 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-dshm\") pod \"sub-test-model-980b82-kserve-5968685f87-r92jn\" (UID: \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\") " pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:06:49.557980 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.557962 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa48f41c-68f1-4f33-838b-33f0a1361dd3-tls-certs\") pod \"sub-test-model-980b82-kserve-5968685f87-r92jn\" (UID: \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\") " pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:06:49.558079 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.558055 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f2sc\" (UniqueName: \"kubernetes.io/projected/aa48f41c-68f1-4f33-838b-33f0a1361dd3-kube-api-access-9f2sc\") pod \"sub-test-model-980b82-kserve-5968685f87-r92jn\" (UID: \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\") " pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:06:49.558157 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.558131 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-home\") pod \"sub-test-model-980b82-kserve-5968685f87-r92jn\" (UID: \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\") " pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:06:49.579909 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.579875 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-980b82\"/\"sub-test-model-980b82-kserve-self-signed-certs\"" Nov 03 00:06:49.579909 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.579880 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-980b82\"/\"default-dockercfg-vqkvp\"" Nov 03 00:06:49.659427 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.659379 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-dshm\") pod \"sub-test-model-980b82-kserve-5968685f87-r92jn\" (UID: \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\") " pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:06:49.659656 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.659436 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa48f41c-68f1-4f33-838b-33f0a1361dd3-tls-certs\") pod \"sub-test-model-980b82-kserve-5968685f87-r92jn\" (UID: \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\") " pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:06:49.659656 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.659471 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f2sc\" (UniqueName: \"kubernetes.io/projected/aa48f41c-68f1-4f33-838b-33f0a1361dd3-kube-api-access-9f2sc\") pod \"sub-test-model-980b82-kserve-5968685f87-r92jn\" (UID: \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\") " pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:06:49.659656 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.659550 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-home\") pod \"sub-test-model-980b82-kserve-5968685f87-r92jn\" (UID: \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\") " pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:06:49.659845 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.659656 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-tmp-dir\") pod \"sub-test-model-980b82-kserve-5968685f87-r92jn\" (UID: \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\") " pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:06:49.659845 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.659704 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-model-cache\") pod \"sub-test-model-980b82-kserve-5968685f87-r92jn\" (UID: \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\") " pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:06:49.660284 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.660129 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-tmp-dir\") pod \"sub-test-model-980b82-kserve-5968685f87-r92jn\" (UID: \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\") " pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:06:49.660284 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.660157 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-home\") pod \"sub-test-model-980b82-kserve-5968685f87-r92jn\" (UID: \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\") " pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:06:49.660284 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.660195 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-model-cache\") pod \"sub-test-model-980b82-kserve-5968685f87-r92jn\" (UID: \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\") " pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:06:49.661888 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.661856 5114 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:06:49.662017 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.661924 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" podUID="65a85aec-06e9-4952-9a16-055faf8cf19b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:06:49.662354 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.662332 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-dshm\") pod \"sub-test-model-980b82-kserve-5968685f87-r92jn\" (UID: \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\") " pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:06:49.662441 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.662402 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa48f41c-68f1-4f33-838b-33f0a1361dd3-tls-certs\") pod \"sub-test-model-980b82-kserve-5968685f87-r92jn\" (UID: \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\") " pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:06:49.696562 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.696523 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt"] Nov 03 00:06:49.697228 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.697189 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" podUID="bc58f9ea-5fa3-44d0-975f-1fed35f2463b" containerName="istio-proxy" containerID="cri-o://0cffd097a2334d4404d9c42a08ba898c8eb2adf70e5dc45640030e365ca82cda" gracePeriod=30 Nov 03 00:06:49.700402 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.700316 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r"] Nov 03 00:06:49.700724 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.700659 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" podUID="ee3fec3e-ad31-4633-b65f-6c97883dea00" containerName="istio-proxy" containerID="cri-o://b84ad2e850d86383d6082fcd1ce0feea0f1268be615bb2f81a26f99a1b2c0fbf" gracePeriod=30 Nov 03 00:06:49.703921 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.703891 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655"] Nov 03 00:06:49.704230 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.704200 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" podUID="7701f578-a649-4f37-956b-aa1189848710" containerName="istio-proxy" containerID="cri-o://7c76cb00678d5ebb455a7e54711c4ed9dfe626a0ff39c851bb0ac61e5190db4a" gracePeriod=30 Nov 03 00:06:49.707388 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.707339 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v"] Nov 03 00:06:49.707649 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.707620 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" podUID="196e5e10-3ef2-4ade-bd72-042921c2f660" containerName="istio-proxy" containerID="cri-o://0b8f999a21cf11a85b0ae39f1c0667c75bc787a3b9715b65cc4d06ae7c503b1a" gracePeriod=30 Nov 03 00:06:49.726782 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.726704 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f2sc\" (UniqueName: \"kubernetes.io/projected/aa48f41c-68f1-4f33-838b-33f0a1361dd3-kube-api-access-9f2sc\") pod \"sub-test-model-980b82-kserve-5968685f87-r92jn\" (UID: \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\") " pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:06:49.782905 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:49.782867 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:06:50.169718 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:50.169691 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn"] Nov 03 00:06:50.171814 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:06:50.171783 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa48f41c_68f1_4f33_838b_33f0a1361dd3.slice/crio-75a94e269951bf1288efb548135177d2553f38fe66ef8c174fa6c28b772c6918 WatchSource:0}: Error finding container 75a94e269951bf1288efb548135177d2553f38fe66ef8c174fa6c28b772c6918: Status 404 returned error can't find the container with id 75a94e269951bf1288efb548135177d2553f38fe66ef8c174fa6c28b772c6918 Nov 03 00:06:50.195547 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:50.195516 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" event={"ID":"aa48f41c-68f1-4f33-838b-33f0a1361dd3","Type":"ContainerStarted","Data":"75a94e269951bf1288efb548135177d2553f38fe66ef8c174fa6c28b772c6918"} Nov 03 00:06:51.204101 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:51.204057 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" event={"ID":"aa48f41c-68f1-4f33-838b-33f0a1361dd3","Type":"ContainerStarted","Data":"fae0955ff4a7bb280b991ed4133bef4782456084f3e9021b444f1511c9229d80"} Nov 03 00:06:51.321572 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:51.321466 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" podStartSLOduration=2.321450319 podStartE2EDuration="2.321450319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:06:51.320072428 +0000 UTC m=+4686.673105806" watchObservedRunningTime="2026-11-03 00:06:51.321450319 +0000 UTC m=+4686.674483755" Nov 03 00:06:51.783986 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:51.783949 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:06:52.352612 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:52.352577 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd12d8e-sqz2s"] Nov 03 00:06:52.356605 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:52.356582 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd12d8e-sqz2s" Nov 03 00:06:52.410769 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:52.410579 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:06:52.460155 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:52.460096 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 03 00:06:52.462888 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:52.462850 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd12d8e-sqz2s"] Nov 03 00:06:52.495103 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:52.495063 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmz5j\" (UniqueName: \"kubernetes.io/projected/a4b5c598-38d9-445d-97a0-724c1a83ef02-kube-api-access-pmz5j\") pod \"maas-api-revoke-sub-bdd12d8e-sqz2s\" (UID: \"a4b5c598-38d9-445d-97a0-724c1a83ef02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd12d8e-sqz2s" Nov 03 00:06:52.495103 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:52.495119 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4b5c598-38d9-445d-97a0-724c1a83ef02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdd12d8e-sqz2s\" (UID: \"a4b5c598-38d9-445d-97a0-724c1a83ef02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd12d8e-sqz2s" Nov 03 00:06:52.495343 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:52.495189 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4b5c598-38d9-445d-97a0-724c1a83ef02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdd12d8e-sqz2s\" (UID: \"a4b5c598-38d9-445d-97a0-724c1a83ef02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd12d8e-sqz2s" Nov 03 00:06:52.597029 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:52.596974 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmz5j\" (UniqueName: \"kubernetes.io/projected/a4b5c598-38d9-445d-97a0-724c1a83ef02-kube-api-access-pmz5j\") pod \"maas-api-revoke-sub-bdd12d8e-sqz2s\" (UID: \"a4b5c598-38d9-445d-97a0-724c1a83ef02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd12d8e-sqz2s" Nov 03 00:06:52.597268 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:52.597056 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4b5c598-38d9-445d-97a0-724c1a83ef02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdd12d8e-sqz2s\" (UID: \"a4b5c598-38d9-445d-97a0-724c1a83ef02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd12d8e-sqz2s" Nov 03 00:06:52.597268 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:52.597097 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4b5c598-38d9-445d-97a0-724c1a83ef02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdd12d8e-sqz2s\" (UID: \"a4b5c598-38d9-445d-97a0-724c1a83ef02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd12d8e-sqz2s" Nov 03 00:06:52.598034 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:52.597989 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4b5c598-38d9-445d-97a0-724c1a83ef02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdd12d8e-sqz2s\" (UID: \"a4b5c598-38d9-445d-97a0-724c1a83ef02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd12d8e-sqz2s" Nov 03 00:06:52.601351 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:52.601316 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4b5c598-38d9-445d-97a0-724c1a83ef02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdd12d8e-sqz2s\" (UID: \"a4b5c598-38d9-445d-97a0-724c1a83ef02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd12d8e-sqz2s" Nov 03 00:06:52.661157 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:52.661079 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmz5j\" (UniqueName: \"kubernetes.io/projected/a4b5c598-38d9-445d-97a0-724c1a83ef02-kube-api-access-pmz5j\") pod \"maas-api-revoke-sub-bdd12d8e-sqz2s\" (UID: \"a4b5c598-38d9-445d-97a0-724c1a83ef02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd12d8e-sqz2s" Nov 03 00:06:52.671999 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:52.671711 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd12d8e-sqz2s" Nov 03 00:06:53.050782 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:53.050756 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd12d8e-sqz2s"] Nov 03 00:06:53.053124 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:06:53.053081 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b5c598_38d9_445d_97a0_724c1a83ef02.slice/crio-c3e3d5e748ebc592f89272bcd2e3109ca4016eb6cc7ba28f5c479ca75f996269 WatchSource:0}: Error finding container c3e3d5e748ebc592f89272bcd2e3109ca4016eb6cc7ba28f5c479ca75f996269: Status 404 returned error can't find the container with id c3e3d5e748ebc592f89272bcd2e3109ca4016eb6cc7ba28f5c479ca75f996269 Nov 03 00:06:53.217200 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:53.217164 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd12d8e-sqz2s" event={"ID":"a4b5c598-38d9-445d-97a0-724c1a83ef02","Type":"ContainerStarted","Data":"c3e3d5e748ebc592f89272bcd2e3109ca4016eb6cc7ba28f5c479ca75f996269"} Nov 03 00:06:53.908378 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:53.908339 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-fr99v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:06:53.908824 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:53.908409 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" podUID="196e5e10-3ef2-4ade-bd72-042921c2f660" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:06:53.973135 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:53.973087 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tp42r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:06:53.973304 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:53.973158 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" podUID="ee3fec3e-ad31-4633-b65f-6c97883dea00" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:06:54.123344 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.123305 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4wgnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:06:54.123643 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.123368 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" podUID="bc58f9ea-5fa3-44d0-975f-1fed35f2463b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:06:54.225610 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.225573 5114 generic.go:356] "Generic (PLEG): container finished" podID="a4b5c598-38d9-445d-97a0-724c1a83ef02" containerID="b3e2972caf4353ad746df73eed712002b4cc57dd1380049ca70f2d1b0316b82d" exitCode=0 Nov 03 00:06:54.225827 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.225692 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd12d8e-sqz2s" event={"ID":"a4b5c598-38d9-445d-97a0-724c1a83ef02","Type":"ContainerDied","Data":"b3e2972caf4353ad746df73eed712002b4cc57dd1380049ca70f2d1b0316b82d"} Nov 03 00:06:54.227882 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.227853 5114 generic.go:356] "Generic (PLEG): container finished" podID="65a85aec-06e9-4952-9a16-055faf8cf19b" containerID="0152612b60723f99b5b0489177ddb16ef9125db6c9d9138d8d1e082943bd2dc6" exitCode=0 Nov 03 00:06:54.228035 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.227929 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" event={"ID":"65a85aec-06e9-4952-9a16-055faf8cf19b","Type":"ContainerDied","Data":"0152612b60723f99b5b0489177ddb16ef9125db6c9d9138d8d1e082943bd2dc6"} Nov 03 00:06:54.272058 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.272029 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:54.402642 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.402544 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gd655 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:06:54.402642 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.402617 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" podUID="7701f578-a649-4f37-956b-aa1189848710" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:06:54.418104 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.418001 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-workload-socket\") pod \"65a85aec-06e9-4952-9a16-055faf8cf19b\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " Nov 03 00:06:54.418104 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.418097 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-envoy\") pod \"65a85aec-06e9-4952-9a16-055faf8cf19b\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " Nov 03 00:06:54.418353 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.418149 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-workload-certs\") pod \"65a85aec-06e9-4952-9a16-055faf8cf19b\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " Nov 03 00:06:54.418353 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.418180 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65a85aec-06e9-4952-9a16-055faf8cf19b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a85aec-06e9-4952-9a16-055faf8cf19b-istiod-ca-cert\") pod \"65a85aec-06e9-4952-9a16-055faf8cf19b\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " Nov 03 00:06:54.418353 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.418215 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-credential-socket\") pod \"65a85aec-06e9-4952-9a16-055faf8cf19b\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " Nov 03 00:06:54.418353 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.418243 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-podinfo\") pod \"65a85aec-06e9-4952-9a16-055faf8cf19b\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " Nov 03 00:06:54.418353 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.418300 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-token\" (UniqueName: \"kubernetes.io/projected/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-token\") pod \"65a85aec-06e9-4952-9a16-055faf8cf19b\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " Nov 03 00:06:54.418353 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.418236 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-workload-socket" pod "65a85aec-06e9-4952-9a16-055faf8cf19b" (UID: "65a85aec-06e9-4952-9a16-055faf8cf19b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:54.418950 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.418902 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-credential-socket" pod "65a85aec-06e9-4952-9a16-055faf8cf19b" (UID: "65a85aec-06e9-4952-9a16-055faf8cf19b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:54.419079 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.418971 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-data" pod "65a85aec-06e9-4952-9a16-055faf8cf19b" (UID: "65a85aec-06e9-4952-9a16-055faf8cf19b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:54.419079 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.419029 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-data\") pod \"65a85aec-06e9-4952-9a16-055faf8cf19b\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " Nov 03 00:06:54.419201 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.419132 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65a85aec-06e9-4952-9a16-055faf8cf19b-kube-api-access-vmjkm\" (UniqueName: \"kubernetes.io/projected/65a85aec-06e9-4952-9a16-055faf8cf19b-kube-api-access-vmjkm\") pod \"65a85aec-06e9-4952-9a16-055faf8cf19b\" (UID: \"65a85aec-06e9-4952-9a16-055faf8cf19b\") " Nov 03 00:06:54.419201 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.419155 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65a85aec-06e9-4952-9a16-055faf8cf19b-istiod-ca-cert" pod "65a85aec-06e9-4952-9a16-055faf8cf19b" (UID: "65a85aec-06e9-4952-9a16-055faf8cf19b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:54.419467 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.419429 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-workload-certs" pod "65a85aec-06e9-4952-9a16-055faf8cf19b" (UID: "65a85aec-06e9-4952-9a16-055faf8cf19b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:54.423841 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.421516 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-envoy" pod "65a85aec-06e9-4952-9a16-055faf8cf19b" (UID: "65a85aec-06e9-4952-9a16-055faf8cf19b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:54.424023 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.423967 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:54.424023 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.423995 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:54.424023 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.424011 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:54.424206 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.424033 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a85aec-06e9-4952-9a16-055faf8cf19b-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:54.424206 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.424048 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:54.424206 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.424063 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:54.434996 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.434859 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-podinfo" pod "65a85aec-06e9-4952-9a16-055faf8cf19b" (UID: "65a85aec-06e9-4952-9a16-055faf8cf19b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:06:54.436091 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.436052 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-token" pod "65a85aec-06e9-4952-9a16-055faf8cf19b" (UID: "65a85aec-06e9-4952-9a16-055faf8cf19b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:54.436214 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.436097 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a85aec-06e9-4952-9a16-055faf8cf19b-kube-api-access-vmjkm" pod "65a85aec-06e9-4952-9a16-055faf8cf19b" (UID: "65a85aec-06e9-4952-9a16-055faf8cf19b"). InnerVolumeSpecName "kube-api-access-vmjkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:54.525045 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.525007 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:54.525045 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.525039 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a85aec-06e9-4952-9a16-055faf8cf19b-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:54.525045 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.525049 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmjkm\" (UniqueName: \"kubernetes.io/projected/65a85aec-06e9-4952-9a16-055faf8cf19b-kube-api-access-vmjkm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:54.975238 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:54.975147 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 03 00:06:55.030410 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.030383 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 03 00:06:55.034171 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.034143 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 03 00:06:55.133215 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.133175 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7701f578-a649-4f37-956b-aa1189848710-istio-token\" (UniqueName: \"kubernetes.io/projected/7701f578-a649-4f37-956b-aa1189848710-istio-token\") pod \"7701f578-a649-4f37-956b-aa1189848710\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " Nov 03 00:06:55.133423 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.133224 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-data\") pod \"ee3fec3e-ad31-4633-b65f-6c97883dea00\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " Nov 03 00:06:55.133423 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.133261 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7701f578-a649-4f37-956b-aa1189848710-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7701f578-a649-4f37-956b-aa1189848710-istiod-ca-cert\") pod \"7701f578-a649-4f37-956b-aa1189848710\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " Nov 03 00:06:55.133423 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.133303 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-kube-api-access-9z5x7\" (UniqueName: \"kubernetes.io/projected/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-kube-api-access-9z5x7\") pod \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " Nov 03 00:06:55.133423 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.133331 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7701f578-a649-4f37-956b-aa1189848710-kube-api-access-g6g5s\" (UniqueName: \"kubernetes.io/projected/7701f578-a649-4f37-956b-aa1189848710-kube-api-access-g6g5s\") pod \"7701f578-a649-4f37-956b-aa1189848710\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " Nov 03 00:06:55.133423 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.133387 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-workload-certs\") pod \"7701f578-a649-4f37-956b-aa1189848710\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " Nov 03 00:06:55.133738 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.133430 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-token\" (UniqueName: \"kubernetes.io/projected/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-token\") pod \"ee3fec3e-ad31-4633-b65f-6c97883dea00\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " Nov 03 00:06:55.133738 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.133458 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-token\" (UniqueName: \"kubernetes.io/projected/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-token\") pod \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " Nov 03 00:06:55.133738 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.133528 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-workload-certs\") pod \"ee3fec3e-ad31-4633-b65f-6c97883dea00\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " Nov 03 00:06:55.133738 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.133571 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-credential-socket\") pod \"ee3fec3e-ad31-4633-b65f-6c97883dea00\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " Nov 03 00:06:55.133738 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.133617 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-istio-data\") pod \"7701f578-a649-4f37-956b-aa1189848710\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " Nov 03 00:06:55.133738 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.133627 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-data" pod "ee3fec3e-ad31-4633-b65f-6c97883dea00" (UID: "ee3fec3e-ad31-4633-b65f-6c97883dea00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:55.133738 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.133654 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-istio-envoy\") pod \"7701f578-a649-4f37-956b-aa1189848710\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " Nov 03 00:06:55.134142 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.133744 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3fec3e-ad31-4633-b65f-6c97883dea00-kube-api-access-6ljz4\" (UniqueName: \"kubernetes.io/projected/ee3fec3e-ad31-4633-b65f-6c97883dea00-kube-api-access-6ljz4\") pod \"ee3fec3e-ad31-4633-b65f-6c97883dea00\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " Nov 03 00:06:55.134142 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.133808 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7701f578-a649-4f37-956b-aa1189848710-istiod-ca-cert" pod "7701f578-a649-4f37-956b-aa1189848710" (UID: "7701f578-a649-4f37-956b-aa1189848710"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:55.134142 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.133818 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-workload-socket\") pod \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " Nov 03 00:06:55.134142 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.133864 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7701f578-a649-4f37-956b-aa1189848710-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7701f578-a649-4f37-956b-aa1189848710-istio-podinfo\") pod \"7701f578-a649-4f37-956b-aa1189848710\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " Nov 03 00:06:55.134142 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.133911 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-podinfo\") pod \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " Nov 03 00:06:55.134142 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.133956 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-workload-socket\") pod \"7701f578-a649-4f37-956b-aa1189848710\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " Nov 03 00:06:55.134142 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.133990 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-workload-socket\") pod \"ee3fec3e-ad31-4633-b65f-6c97883dea00\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " Nov 03 00:06:55.134142 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.134027 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istiod-ca-cert\") pod \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " Nov 03 00:06:55.134142 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.134024 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-workload-certs" pod "7701f578-a649-4f37-956b-aa1189848710" (UID: "7701f578-a649-4f37-956b-aa1189848710"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:55.134142 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.134062 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-data\") pod \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " Nov 03 00:06:55.134142 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.134072 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-credential-socket" pod "ee3fec3e-ad31-4633-b65f-6c97883dea00" (UID: "ee3fec3e-ad31-4633-b65f-6c97883dea00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:55.134142 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.134096 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-credential-socket\") pod \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " Nov 03 00:06:55.134142 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.134138 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-credential-socket\") pod \"7701f578-a649-4f37-956b-aa1189848710\" (UID: \"7701f578-a649-4f37-956b-aa1189848710\") " Nov 03 00:06:55.135044 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.134178 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee3fec3e-ad31-4633-b65f-6c97883dea00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3fec3e-ad31-4633-b65f-6c97883dea00-istiod-ca-cert\") pod \"ee3fec3e-ad31-4633-b65f-6c97883dea00\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " Nov 03 00:06:55.135044 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.134231 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-podinfo\") pod \"ee3fec3e-ad31-4633-b65f-6c97883dea00\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " Nov 03 00:06:55.135044 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.134260 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-workload-certs" pod "ee3fec3e-ad31-4633-b65f-6c97883dea00" (UID: "ee3fec3e-ad31-4633-b65f-6c97883dea00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:55.135044 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.134301 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-envoy\") pod \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " Nov 03 00:06:55.135044 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.134341 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-envoy\") pod \"ee3fec3e-ad31-4633-b65f-6c97883dea00\" (UID: \"ee3fec3e-ad31-4633-b65f-6c97883dea00\") " Nov 03 00:06:55.135044 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.134383 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-workload-certs\") pod \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\" (UID: \"bc58f9ea-5fa3-44d0-975f-1fed35f2463b\") " Nov 03 00:06:55.135044 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.134861 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-workload-socket" pod "bc58f9ea-5fa3-44d0-975f-1fed35f2463b" (UID: "bc58f9ea-5fa3-44d0-975f-1fed35f2463b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:55.135044 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.134874 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-istio-data" pod "7701f578-a649-4f37-956b-aa1189848710" (UID: "7701f578-a649-4f37-956b-aa1189848710"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:55.135044 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.135030 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.136006 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.135053 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7701f578-a649-4f37-956b-aa1189848710-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.136006 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.135070 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.136006 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.135076 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-workload-certs" pod "bc58f9ea-5fa3-44d0-975f-1fed35f2463b" (UID: "bc58f9ea-5fa3-44d0-975f-1fed35f2463b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:55.136006 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.135087 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.136006 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.135103 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.136006 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.135104 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-credential-socket" pod "bc58f9ea-5fa3-44d0-975f-1fed35f2463b" (UID: "bc58f9ea-5fa3-44d0-975f-1fed35f2463b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:55.136006 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.135116 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.136006 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.135132 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.136006 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.135338 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-credential-socket" pod "7701f578-a649-4f37-956b-aa1189848710" (UID: "7701f578-a649-4f37-956b-aa1189848710"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:55.136006 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.135527 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-workload-socket" pod "7701f578-a649-4f37-956b-aa1189848710" (UID: "7701f578-a649-4f37-956b-aa1189848710"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:55.136006 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.135747 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-workload-socket" pod "ee3fec3e-ad31-4633-b65f-6c97883dea00" (UID: "ee3fec3e-ad31-4633-b65f-6c97883dea00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:55.136006 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.135890 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee3fec3e-ad31-4633-b65f-6c97883dea00-istiod-ca-cert" pod "ee3fec3e-ad31-4633-b65f-6c97883dea00" (UID: "ee3fec3e-ad31-4633-b65f-6c97883dea00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:55.136858 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.136169 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-data" pod "bc58f9ea-5fa3-44d0-975f-1fed35f2463b" (UID: "bc58f9ea-5fa3-44d0-975f-1fed35f2463b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:55.136858 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.136533 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istiod-ca-cert" pod "bc58f9ea-5fa3-44d0-975f-1fed35f2463b" (UID: "bc58f9ea-5fa3-44d0-975f-1fed35f2463b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:55.137055 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.137028 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-istio-envoy" pod "7701f578-a649-4f37-956b-aa1189848710" (UID: "7701f578-a649-4f37-956b-aa1189848710"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:55.137407 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.137356 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-podinfo" pod "bc58f9ea-5fa3-44d0-975f-1fed35f2463b" (UID: "bc58f9ea-5fa3-44d0-975f-1fed35f2463b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:06:55.137688 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.137661 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-envoy" pod "bc58f9ea-5fa3-44d0-975f-1fed35f2463b" (UID: "bc58f9ea-5fa3-44d0-975f-1fed35f2463b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:55.138620 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.138591 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-token" pod "bc58f9ea-5fa3-44d0-975f-1fed35f2463b" (UID: "bc58f9ea-5fa3-44d0-975f-1fed35f2463b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:55.138737 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.138632 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-kube-api-access-9z5x7" pod "bc58f9ea-5fa3-44d0-975f-1fed35f2463b" (UID: "bc58f9ea-5fa3-44d0-975f-1fed35f2463b"). InnerVolumeSpecName "kube-api-access-9z5x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:55.139230 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.139209 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-envoy" pod "ee3fec3e-ad31-4633-b65f-6c97883dea00" (UID: "ee3fec3e-ad31-4633-b65f-6c97883dea00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:55.139230 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.139208 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-token" pod "ee3fec3e-ad31-4633-b65f-6c97883dea00" (UID: "ee3fec3e-ad31-4633-b65f-6c97883dea00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:55.139597 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.139359 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7701f578-a649-4f37-956b-aa1189848710-istio-token" pod "7701f578-a649-4f37-956b-aa1189848710" (UID: "7701f578-a649-4f37-956b-aa1189848710"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:55.139597 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.139372 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7701f578-a649-4f37-956b-aa1189848710-kube-api-access-g6g5s" pod "7701f578-a649-4f37-956b-aa1189848710" (UID: "7701f578-a649-4f37-956b-aa1189848710"). InnerVolumeSpecName "kube-api-access-g6g5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:55.139597 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.139541 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-podinfo" pod "ee3fec3e-ad31-4633-b65f-6c97883dea00" (UID: "ee3fec3e-ad31-4633-b65f-6c97883dea00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:06:55.140200 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.140171 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3fec3e-ad31-4633-b65f-6c97883dea00-kube-api-access-6ljz4" pod "ee3fec3e-ad31-4633-b65f-6c97883dea00" (UID: "ee3fec3e-ad31-4633-b65f-6c97883dea00"). InnerVolumeSpecName "kube-api-access-6ljz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:55.140881 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.140861 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7701f578-a649-4f37-956b-aa1189848710-istio-podinfo" pod "7701f578-a649-4f37-956b-aa1189848710" (UID: "7701f578-a649-4f37-956b-aa1189848710"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:06:55.236079 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.235936 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.236079 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.235969 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.236079 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.235979 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.236079 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.235991 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7701f578-a649-4f37-956b-aa1189848710-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.236079 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.236001 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z5x7\" (UniqueName: \"kubernetes.io/projected/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-kube-api-access-9z5x7\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.236079 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.236011 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6g5s\" (UniqueName: \"kubernetes.io/projected/7701f578-a649-4f37-956b-aa1189848710-kube-api-access-g6g5s\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.236079 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.236021 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.236079 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.236031 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.236079 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.236047 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.236079 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.236055 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ljz4\" (UniqueName: \"kubernetes.io/projected/ee3fec3e-ad31-4633-b65f-6c97883dea00-kube-api-access-6ljz4\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.236079 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.236063 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7701f578-a649-4f37-956b-aa1189848710-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.236079 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.236072 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.236079 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.236084 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.236079 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.236096 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3fec3e-ad31-4633-b65f-6c97883dea00-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.236753 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.236105 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.236753 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.236117 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.236753 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.236131 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc58f9ea-5fa3-44d0-975f-1fed35f2463b-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.236753 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.236139 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7701f578-a649-4f37-956b-aa1189848710-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.236753 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.236149 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3fec3e-ad31-4633-b65f-6c97883dea00-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.236753 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.236157 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3fec3e-ad31-4633-b65f-6c97883dea00-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.236753 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.236474 5114 generic.go:356] "Generic (PLEG): container finished" podID="7701f578-a649-4f37-956b-aa1189848710" containerID="7c76cb00678d5ebb455a7e54711c4ed9dfe626a0ff39c851bb0ac61e5190db4a" exitCode=0 Nov 03 00:06:55.236753 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.236575 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" event={"ID":"7701f578-a649-4f37-956b-aa1189848710","Type":"ContainerDied","Data":"7c76cb00678d5ebb455a7e54711c4ed9dfe626a0ff39c851bb0ac61e5190db4a"} Nov 03 00:06:55.236753 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.236623 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" event={"ID":"7701f578-a649-4f37-956b-aa1189848710","Type":"ContainerDied","Data":"8a6e3f2f7eb2c19bae79a758e5a436f1874bed131fcd94f53f73a049ed352616"} Nov 03 00:06:55.236753 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.236643 5114 scope.go:122] "RemoveContainer" containerID="7c76cb00678d5ebb455a7e54711c4ed9dfe626a0ff39c851bb0ac61e5190db4a" Nov 03 00:06:55.237078 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.236777 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655" Nov 03 00:06:55.239286 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.239258 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" Nov 03 00:06:55.239407 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.239263 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw" event={"ID":"65a85aec-06e9-4952-9a16-055faf8cf19b","Type":"ContainerDied","Data":"574480419f5959b194aa57e598a59fdfc16ebe63d3c5cc8a3d35e4ce4ba83f6f"} Nov 03 00:06:55.243171 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.243091 5114 generic.go:356] "Generic (PLEG): container finished" podID="196e5e10-3ef2-4ade-bd72-042921c2f660" containerID="0b8f999a21cf11a85b0ae39f1c0667c75bc787a3b9715b65cc4d06ae7c503b1a" exitCode=0 Nov 03 00:06:55.243347 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.243322 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" event={"ID":"196e5e10-3ef2-4ade-bd72-042921c2f660","Type":"ContainerDied","Data":"0b8f999a21cf11a85b0ae39f1c0667c75bc787a3b9715b65cc4d06ae7c503b1a"} Nov 03 00:06:55.245649 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.245615 5114 generic.go:356] "Generic (PLEG): container finished" podID="bc58f9ea-5fa3-44d0-975f-1fed35f2463b" containerID="0cffd097a2334d4404d9c42a08ba898c8eb2adf70e5dc45640030e365ca82cda" exitCode=0 Nov 03 00:06:55.245792 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.245777 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" Nov 03 00:06:55.246428 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.246358 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" event={"ID":"bc58f9ea-5fa3-44d0-975f-1fed35f2463b","Type":"ContainerDied","Data":"0cffd097a2334d4404d9c42a08ba898c8eb2adf70e5dc45640030e365ca82cda"} Nov 03 00:06:55.246923 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.246746 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt" event={"ID":"bc58f9ea-5fa3-44d0-975f-1fed35f2463b","Type":"ContainerDied","Data":"7e00f9c5786bc6c2598ca98c99e5d5fe73e76d96a86b29c85ed8da9b55ce552f"} Nov 03 00:06:55.248767 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.248330 5114 generic.go:356] "Generic (PLEG): container finished" podID="ee3fec3e-ad31-4633-b65f-6c97883dea00" containerID="b84ad2e850d86383d6082fcd1ce0feea0f1268be615bb2f81a26f99a1b2c0fbf" exitCode=0 Nov 03 00:06:55.248767 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.248619 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" Nov 03 00:06:55.252719 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.252581 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" event={"ID":"ee3fec3e-ad31-4633-b65f-6c97883dea00","Type":"ContainerDied","Data":"b84ad2e850d86383d6082fcd1ce0feea0f1268be615bb2f81a26f99a1b2c0fbf"} Nov 03 00:06:55.252719 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.252645 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r" event={"ID":"ee3fec3e-ad31-4633-b65f-6c97883dea00","Type":"ContainerDied","Data":"163441c5fbaefc7c246ef96685c060c1abf7baf72fd0a0a80b685287549dda98"} Nov 03 00:06:55.255419 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.255389 5114 scope.go:122] "RemoveContainer" containerID="7c76cb00678d5ebb455a7e54711c4ed9dfe626a0ff39c851bb0ac61e5190db4a" Nov 03 00:06:55.256900 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:06:55.255883 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c76cb00678d5ebb455a7e54711c4ed9dfe626a0ff39c851bb0ac61e5190db4a\": container with ID starting with 7c76cb00678d5ebb455a7e54711c4ed9dfe626a0ff39c851bb0ac61e5190db4a not found: ID does not exist" containerID="7c76cb00678d5ebb455a7e54711c4ed9dfe626a0ff39c851bb0ac61e5190db4a" Nov 03 00:06:55.256900 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.255920 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c76cb00678d5ebb455a7e54711c4ed9dfe626a0ff39c851bb0ac61e5190db4a"} err="failed to get container status \"7c76cb00678d5ebb455a7e54711c4ed9dfe626a0ff39c851bb0ac61e5190db4a\": rpc error: code = NotFound desc = could not find container \"7c76cb00678d5ebb455a7e54711c4ed9dfe626a0ff39c851bb0ac61e5190db4a\": container with ID starting with 7c76cb00678d5ebb455a7e54711c4ed9dfe626a0ff39c851bb0ac61e5190db4a not found: ID does not exist" Nov 03 00:06:55.256900 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.255944 5114 scope.go:122] "RemoveContainer" containerID="0152612b60723f99b5b0489177ddb16ef9125db6c9d9138d8d1e082943bd2dc6" Nov 03 00:06:55.274122 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.274098 5114 scope.go:122] "RemoveContainer" containerID="0cffd097a2334d4404d9c42a08ba898c8eb2adf70e5dc45640030e365ca82cda" Nov 03 00:06:55.286447 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.286422 5114 scope.go:122] "RemoveContainer" containerID="0cffd097a2334d4404d9c42a08ba898c8eb2adf70e5dc45640030e365ca82cda" Nov 03 00:06:55.286962 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:06:55.286934 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cffd097a2334d4404d9c42a08ba898c8eb2adf70e5dc45640030e365ca82cda\": container with ID starting with 0cffd097a2334d4404d9c42a08ba898c8eb2adf70e5dc45640030e365ca82cda not found: ID does not exist" containerID="0cffd097a2334d4404d9c42a08ba898c8eb2adf70e5dc45640030e365ca82cda" Nov 03 00:06:55.287055 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.286979 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cffd097a2334d4404d9c42a08ba898c8eb2adf70e5dc45640030e365ca82cda"} err="failed to get container status \"0cffd097a2334d4404d9c42a08ba898c8eb2adf70e5dc45640030e365ca82cda\": rpc error: code = NotFound desc = could not find container \"0cffd097a2334d4404d9c42a08ba898c8eb2adf70e5dc45640030e365ca82cda\": container with ID starting with 0cffd097a2334d4404d9c42a08ba898c8eb2adf70e5dc45640030e365ca82cda not found: ID does not exist" Nov 03 00:06:55.287055 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.287013 5114 scope.go:122] "RemoveContainer" containerID="b84ad2e850d86383d6082fcd1ce0feea0f1268be615bb2f81a26f99a1b2c0fbf" Nov 03 00:06:55.306745 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.303639 5114 scope.go:122] "RemoveContainer" containerID="b84ad2e850d86383d6082fcd1ce0feea0f1268be615bb2f81a26f99a1b2c0fbf" Nov 03 00:06:55.306745 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:06:55.304170 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b84ad2e850d86383d6082fcd1ce0feea0f1268be615bb2f81a26f99a1b2c0fbf\": container with ID starting with b84ad2e850d86383d6082fcd1ce0feea0f1268be615bb2f81a26f99a1b2c0fbf not found: ID does not exist" containerID="b84ad2e850d86383d6082fcd1ce0feea0f1268be615bb2f81a26f99a1b2c0fbf" Nov 03 00:06:55.306745 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.304224 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b84ad2e850d86383d6082fcd1ce0feea0f1268be615bb2f81a26f99a1b2c0fbf"} err="failed to get container status \"b84ad2e850d86383d6082fcd1ce0feea0f1268be615bb2f81a26f99a1b2c0fbf\": rpc error: code = NotFound desc = could not find container \"b84ad2e850d86383d6082fcd1ce0feea0f1268be615bb2f81a26f99a1b2c0fbf\": container with ID starting with b84ad2e850d86383d6082fcd1ce0feea0f1268be615bb2f81a26f99a1b2c0fbf not found: ID does not exist" Nov 03 00:06:55.399470 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.399434 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 03 00:06:55.428595 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.428563 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd12d8e-sqz2s" Nov 03 00:06:55.459722 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.459685 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt"] Nov 03 00:06:55.465781 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.465742 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wgnt"] Nov 03 00:06:55.539536 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.539475 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-workload-certs\") pod \"196e5e10-3ef2-4ade-bd72-042921c2f660\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " Nov 03 00:06:55.539726 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.539549 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-podinfo\") pod \"196e5e10-3ef2-4ade-bd72-042921c2f660\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " Nov 03 00:06:55.539726 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.539579 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-data\") pod \"196e5e10-3ef2-4ade-bd72-042921c2f660\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " Nov 03 00:06:55.539726 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.539619 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4b5c598-38d9-445d-97a0-724c1a83ef02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4b5c598-38d9-445d-97a0-724c1a83ef02-maas-api-openshift-ca\") pod \"a4b5c598-38d9-445d-97a0-724c1a83ef02\" (UID: \"a4b5c598-38d9-445d-97a0-724c1a83ef02\") " Nov 03 00:06:55.539726 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.539696 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-token\" (UniqueName: \"kubernetes.io/projected/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-token\") pod \"196e5e10-3ef2-4ade-bd72-042921c2f660\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " Nov 03 00:06:55.540031 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.539798 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-workload-certs" pod "196e5e10-3ef2-4ade-bd72-042921c2f660" (UID: "196e5e10-3ef2-4ade-bd72-042921c2f660"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:55.540031 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.539813 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/196e5e10-3ef2-4ade-bd72-042921c2f660-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/196e5e10-3ef2-4ade-bd72-042921c2f660-istiod-ca-cert\") pod \"196e5e10-3ef2-4ade-bd72-042921c2f660\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " Nov 03 00:06:55.540031 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.539891 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-data" pod "196e5e10-3ef2-4ade-bd72-042921c2f660" (UID: "196e5e10-3ef2-4ade-bd72-042921c2f660"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:55.540031 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.539933 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-workload-socket\") pod \"196e5e10-3ef2-4ade-bd72-042921c2f660\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " Nov 03 00:06:55.540031 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.540010 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b5c598-38d9-445d-97a0-724c1a83ef02-maas-api-openshift-ca" pod "a4b5c598-38d9-445d-97a0-724c1a83ef02" (UID: "a4b5c598-38d9-445d-97a0-724c1a83ef02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:55.540377 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.540031 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-credential-socket\") pod \"196e5e10-3ef2-4ade-bd72-042921c2f660\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " Nov 03 00:06:55.540377 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.540075 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-envoy\") pod \"196e5e10-3ef2-4ade-bd72-042921c2f660\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " Nov 03 00:06:55.540377 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.540108 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4b5c598-38d9-445d-97a0-724c1a83ef02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4b5c598-38d9-445d-97a0-724c1a83ef02-maas-api-xks-ca\") pod \"a4b5c598-38d9-445d-97a0-724c1a83ef02\" (UID: \"a4b5c598-38d9-445d-97a0-724c1a83ef02\") " Nov 03 00:06:55.540377 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.540136 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4b5c598-38d9-445d-97a0-724c1a83ef02-kube-api-access-pmz5j\" (UniqueName: \"kubernetes.io/projected/a4b5c598-38d9-445d-97a0-724c1a83ef02-kube-api-access-pmz5j\") pod \"a4b5c598-38d9-445d-97a0-724c1a83ef02\" (UID: \"a4b5c598-38d9-445d-97a0-724c1a83ef02\") " Nov 03 00:06:55.540377 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.540167 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/196e5e10-3ef2-4ade-bd72-042921c2f660-kube-api-access-q9mbq\" (UniqueName: \"kubernetes.io/projected/196e5e10-3ef2-4ade-bd72-042921c2f660-kube-api-access-q9mbq\") pod \"196e5e10-3ef2-4ade-bd72-042921c2f660\" (UID: \"196e5e10-3ef2-4ade-bd72-042921c2f660\") " Nov 03 00:06:55.540377 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.540179 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-workload-socket" pod "196e5e10-3ef2-4ade-bd72-042921c2f660" (UID: "196e5e10-3ef2-4ade-bd72-042921c2f660"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:55.540377 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.540186 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/196e5e10-3ef2-4ade-bd72-042921c2f660-istiod-ca-cert" pod "196e5e10-3ef2-4ade-bd72-042921c2f660" (UID: "196e5e10-3ef2-4ade-bd72-042921c2f660"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:55.540377 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.540294 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-credential-socket" pod "196e5e10-3ef2-4ade-bd72-042921c2f660" (UID: "196e5e10-3ef2-4ade-bd72-042921c2f660"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:55.540799 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.540638 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.540799 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.540660 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.540799 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.540675 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4b5c598-38d9-445d-97a0-724c1a83ef02-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.540799 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.540693 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/196e5e10-3ef2-4ade-bd72-042921c2f660-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.540799 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.540708 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.540799 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.540722 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.542955 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.542930 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-envoy" pod "196e5e10-3ef2-4ade-bd72-042921c2f660" (UID: "196e5e10-3ef2-4ade-bd72-042921c2f660"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:55.542955 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.542937 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-podinfo" pod "196e5e10-3ef2-4ade-bd72-042921c2f660" (UID: "196e5e10-3ef2-4ade-bd72-042921c2f660"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:06:55.543085 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.543022 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4b5c598-38d9-445d-97a0-724c1a83ef02-maas-api-xks-ca" pod "a4b5c598-38d9-445d-97a0-724c1a83ef02" (UID: "a4b5c598-38d9-445d-97a0-724c1a83ef02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:55.543144 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.543083 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-token" pod "196e5e10-3ef2-4ade-bd72-042921c2f660" (UID: "196e5e10-3ef2-4ade-bd72-042921c2f660"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:55.543144 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.543102 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b5c598-38d9-445d-97a0-724c1a83ef02-kube-api-access-pmz5j" pod "a4b5c598-38d9-445d-97a0-724c1a83ef02" (UID: "a4b5c598-38d9-445d-97a0-724c1a83ef02"). InnerVolumeSpecName "kube-api-access-pmz5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:55.543238 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.543157 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/196e5e10-3ef2-4ade-bd72-042921c2f660-kube-api-access-q9mbq" pod "196e5e10-3ef2-4ade-bd72-042921c2f660" (UID: "196e5e10-3ef2-4ade-bd72-042921c2f660"). InnerVolumeSpecName "kube-api-access-q9mbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:55.573533 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.573493 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw"] Nov 03 00:06:55.577682 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.577648 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-2blxw"] Nov 03 00:06:55.626922 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.626883 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" podUID="f850c9fa-e500-4772-8454-a5172e446770" containerName="main" containerID="cri-o://2b54f75a1e0f0551c482460ae98ea70b93e7565573be0b69f756a7e4ffcc03b2" gracePeriod=45 Nov 03 00:06:55.641521 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.641459 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.641521 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.641519 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.641521 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.641531 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4b5c598-38d9-445d-97a0-724c1a83ef02-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.641835 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.641541 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmz5j\" (UniqueName: \"kubernetes.io/projected/a4b5c598-38d9-445d-97a0-724c1a83ef02-kube-api-access-pmz5j\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.641835 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.641553 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9mbq\" (UniqueName: \"kubernetes.io/projected/196e5e10-3ef2-4ade-bd72-042921c2f660-kube-api-access-q9mbq\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.641835 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.641562 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/196e5e10-3ef2-4ade-bd72-042921c2f660-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:55.648957 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.648916 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" podUID="fb16fc7d-3def-41dc-91a7-1103ccaf4c35" containerName="main" containerID="cri-o://98de9b33e0b6bccb5280f8d48580c653c28c20f0f4e97c78969b4a852ab2af38" gracePeriod=45 Nov 03 00:06:55.686221 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.686174 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r"] Nov 03 00:06:55.691022 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.690990 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tp42r"] Nov 03 00:06:55.798451 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.798350 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655"] Nov 03 00:06:55.806605 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.806572 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gd655"] Nov 03 00:06:55.929131 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.929100 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 03 00:06:55.945432 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:55.945400 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 03 00:06:56.045814 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.045777 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-home\" (UniqueName: \"kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-home\") pod \"f850c9fa-e500-4772-8454-a5172e446770\" (UID: \"f850c9fa-e500-4772-8454-a5172e446770\") " Nov 03 00:06:56.046333 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.045837 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-dshm\") pod \"f850c9fa-e500-4772-8454-a5172e446770\" (UID: \"f850c9fa-e500-4772-8454-a5172e446770\") " Nov 03 00:06:56.046333 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.045864 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f850c9fa-e500-4772-8454-a5172e446770-kube-api-access-npv27\" (UniqueName: \"kubernetes.io/projected/f850c9fa-e500-4772-8454-a5172e446770-kube-api-access-npv27\") pod \"f850c9fa-e500-4772-8454-a5172e446770\" (UID: \"f850c9fa-e500-4772-8454-a5172e446770\") " Nov 03 00:06:56.046333 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.045898 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-tls-certs\") pod \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\" (UID: \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\") " Nov 03 00:06:56.046333 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.045939 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-home\" (UniqueName: \"kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-home\") pod \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\" (UID: \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\") " Nov 03 00:06:56.046333 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.046106 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-home" pod "f850c9fa-e500-4772-8454-a5172e446770" (UID: "f850c9fa-e500-4772-8454-a5172e446770"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:56.046333 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.046191 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-home" pod "fb16fc7d-3def-41dc-91a7-1103ccaf4c35" (UID: "fb16fc7d-3def-41dc-91a7-1103ccaf4c35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:56.046690 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.046363 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-dshm\") pod \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\" (UID: \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\") " Nov 03 00:06:56.046690 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.046415 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-tmp-dir\") pod \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\" (UID: \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\") " Nov 03 00:06:56.046690 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.046541 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-model-cache\") pod \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\" (UID: \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\") " Nov 03 00:06:56.046690 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.046675 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-kube-api-access-lvrd4\" (UniqueName: \"kubernetes.io/projected/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-kube-api-access-lvrd4\") pod \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\" (UID: \"fb16fc7d-3def-41dc-91a7-1103ccaf4c35\") " Nov 03 00:06:56.046893 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.046707 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-tmp-dir\") pod \"f850c9fa-e500-4772-8454-a5172e446770\" (UID: \"f850c9fa-e500-4772-8454-a5172e446770\") " Nov 03 00:06:56.046893 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.046775 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f850c9fa-e500-4772-8454-a5172e446770-tls-certs\" (UniqueName: \"kubernetes.io/secret/f850c9fa-e500-4772-8454-a5172e446770-tls-certs\") pod \"f850c9fa-e500-4772-8454-a5172e446770\" (UID: \"f850c9fa-e500-4772-8454-a5172e446770\") " Nov 03 00:06:56.046893 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.046805 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-model-cache\") pod \"f850c9fa-e500-4772-8454-a5172e446770\" (UID: \"f850c9fa-e500-4772-8454-a5172e446770\") " Nov 03 00:06:56.047057 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.047033 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-model-cache" pod "fb16fc7d-3def-41dc-91a7-1103ccaf4c35" (UID: "fb16fc7d-3def-41dc-91a7-1103ccaf4c35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:56.047216 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.047198 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:56.047296 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.047224 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:56.047296 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.047226 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-tmp-dir" pod "fb16fc7d-3def-41dc-91a7-1103ccaf4c35" (UID: "fb16fc7d-3def-41dc-91a7-1103ccaf4c35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:56.047296 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.047236 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:56.047465 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.047435 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-tmp-dir" pod "f850c9fa-e500-4772-8454-a5172e446770" (UID: "f850c9fa-e500-4772-8454-a5172e446770"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:56.047953 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.047926 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-model-cache" pod "f850c9fa-e500-4772-8454-a5172e446770" (UID: "f850c9fa-e500-4772-8454-a5172e446770"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:56.049010 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.048939 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-dshm" pod "f850c9fa-e500-4772-8454-a5172e446770" (UID: "f850c9fa-e500-4772-8454-a5172e446770"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:56.049414 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.049389 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f850c9fa-e500-4772-8454-a5172e446770-kube-api-access-npv27" pod "f850c9fa-e500-4772-8454-a5172e446770" (UID: "f850c9fa-e500-4772-8454-a5172e446770"). InnerVolumeSpecName "kube-api-access-npv27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:56.049765 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.049743 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-tls-certs" pod "fb16fc7d-3def-41dc-91a7-1103ccaf4c35" (UID: "fb16fc7d-3def-41dc-91a7-1103ccaf4c35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:56.050167 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.050148 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-dshm" pod "fb16fc7d-3def-41dc-91a7-1103ccaf4c35" (UID: "fb16fc7d-3def-41dc-91a7-1103ccaf4c35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:56.050377 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.050358 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f850c9fa-e500-4772-8454-a5172e446770-tls-certs" pod "f850c9fa-e500-4772-8454-a5172e446770" (UID: "f850c9fa-e500-4772-8454-a5172e446770"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:56.050464 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.050361 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-kube-api-access-lvrd4" pod "fb16fc7d-3def-41dc-91a7-1103ccaf4c35" (UID: "fb16fc7d-3def-41dc-91a7-1103ccaf4c35"). InnerVolumeSpecName "kube-api-access-lvrd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:56.148373 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.148325 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvrd4\" (UniqueName: \"kubernetes.io/projected/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-kube-api-access-lvrd4\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:56.148373 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.148371 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:56.148668 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.148389 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f850c9fa-e500-4772-8454-a5172e446770-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:56.148668 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.148405 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:56.148668 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.148420 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f850c9fa-e500-4772-8454-a5172e446770-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:56.148668 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.148436 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npv27\" (UniqueName: \"kubernetes.io/projected/f850c9fa-e500-4772-8454-a5172e446770-kube-api-access-npv27\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:56.148668 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.148450 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:56.148668 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.148465 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:56.148668 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.148500 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb16fc7d-3def-41dc-91a7-1103ccaf4c35-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:56.258078 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.258035 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" Nov 03 00:06:56.258306 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.258042 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v" event={"ID":"196e5e10-3ef2-4ade-bd72-042921c2f660","Type":"ContainerDied","Data":"1d6510a6288d7b08f4df559b084702fa8e97eba4b48c01a6407091b182053e4d"} Nov 03 00:06:56.258306 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.258195 5114 scope.go:122] "RemoveContainer" containerID="0b8f999a21cf11a85b0ae39f1c0667c75bc787a3b9715b65cc4d06ae7c503b1a" Nov 03 00:06:56.260081 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.260057 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd12d8e-sqz2s" Nov 03 00:06:56.260245 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.260079 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd12d8e-sqz2s" event={"ID":"a4b5c598-38d9-445d-97a0-724c1a83ef02","Type":"ContainerDied","Data":"c3e3d5e748ebc592f89272bcd2e3109ca4016eb6cc7ba28f5c479ca75f996269"} Nov 03 00:06:56.260245 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.260133 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3e3d5e748ebc592f89272bcd2e3109ca4016eb6cc7ba28f5c479ca75f996269" Nov 03 00:06:56.295839 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.292205 5114 generic.go:356] "Generic (PLEG): container finished" podID="fb16fc7d-3def-41dc-91a7-1103ccaf4c35" containerID="98de9b33e0b6bccb5280f8d48580c653c28c20f0f4e97c78969b4a852ab2af38" exitCode=0 Nov 03 00:06:56.295839 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.292343 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" event={"ID":"fb16fc7d-3def-41dc-91a7-1103ccaf4c35","Type":"ContainerDied","Data":"98de9b33e0b6bccb5280f8d48580c653c28c20f0f4e97c78969b4a852ab2af38"} Nov 03 00:06:56.295839 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.292383 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" event={"ID":"fb16fc7d-3def-41dc-91a7-1103ccaf4c35","Type":"ContainerDied","Data":"6590208f65f9f023893795452d3f4023b70e4442012e777e31fdfe6ed74fc28b"} Nov 03 00:06:56.295839 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.292407 5114 scope.go:122] "RemoveContainer" containerID="98de9b33e0b6bccb5280f8d48580c653c28c20f0f4e97c78969b4a852ab2af38" Nov 03 00:06:56.295839 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.292552 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t" Nov 03 00:06:56.317100 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.316905 5114 generic.go:356] "Generic (PLEG): container finished" podID="f850c9fa-e500-4772-8454-a5172e446770" containerID="2b54f75a1e0f0551c482460ae98ea70b93e7565573be0b69f756a7e4ffcc03b2" exitCode=0 Nov 03 00:06:56.317100 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.317018 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" Nov 03 00:06:56.317100 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.317036 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" event={"ID":"f850c9fa-e500-4772-8454-a5172e446770","Type":"ContainerDied","Data":"2b54f75a1e0f0551c482460ae98ea70b93e7565573be0b69f756a7e4ffcc03b2"} Nov 03 00:06:56.317100 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.317083 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd" event={"ID":"f850c9fa-e500-4772-8454-a5172e446770","Type":"ContainerDied","Data":"6a512ac8aec386162951c15273e5687c7e4916153a027ab37d260beb40266fcd"} Nov 03 00:06:56.325730 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.325701 5114 scope.go:122] "RemoveContainer" containerID="98de9b33e0b6bccb5280f8d48580c653c28c20f0f4e97c78969b4a852ab2af38" Nov 03 00:06:56.326321 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:06:56.326182 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98de9b33e0b6bccb5280f8d48580c653c28c20f0f4e97c78969b4a852ab2af38\": container with ID starting with 98de9b33e0b6bccb5280f8d48580c653c28c20f0f4e97c78969b4a852ab2af38 not found: ID does not exist" containerID="98de9b33e0b6bccb5280f8d48580c653c28c20f0f4e97c78969b4a852ab2af38" Nov 03 00:06:56.326321 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.326218 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98de9b33e0b6bccb5280f8d48580c653c28c20f0f4e97c78969b4a852ab2af38"} err="failed to get container status \"98de9b33e0b6bccb5280f8d48580c653c28c20f0f4e97c78969b4a852ab2af38\": rpc error: code = NotFound desc = could not find container \"98de9b33e0b6bccb5280f8d48580c653c28c20f0f4e97c78969b4a852ab2af38\": container with ID starting with 98de9b33e0b6bccb5280f8d48580c653c28c20f0f4e97c78969b4a852ab2af38 not found: ID does not exist" Nov 03 00:06:56.326321 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.326245 5114 scope.go:122] "RemoveContainer" containerID="2b54f75a1e0f0551c482460ae98ea70b93e7565573be0b69f756a7e4ffcc03b2" Nov 03 00:06:56.336782 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.336757 5114 scope.go:122] "RemoveContainer" containerID="2b54f75a1e0f0551c482460ae98ea70b93e7565573be0b69f756a7e4ffcc03b2" Nov 03 00:06:56.337132 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:06:56.337103 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b54f75a1e0f0551c482460ae98ea70b93e7565573be0b69f756a7e4ffcc03b2\": container with ID starting with 2b54f75a1e0f0551c482460ae98ea70b93e7565573be0b69f756a7e4ffcc03b2 not found: ID does not exist" containerID="2b54f75a1e0f0551c482460ae98ea70b93e7565573be0b69f756a7e4ffcc03b2" Nov 03 00:06:56.337219 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.337150 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b54f75a1e0f0551c482460ae98ea70b93e7565573be0b69f756a7e4ffcc03b2"} err="failed to get container status \"2b54f75a1e0f0551c482460ae98ea70b93e7565573be0b69f756a7e4ffcc03b2\": rpc error: code = NotFound desc = could not find container \"2b54f75a1e0f0551c482460ae98ea70b93e7565573be0b69f756a7e4ffcc03b2\": container with ID starting with 2b54f75a1e0f0551c482460ae98ea70b93e7565573be0b69f756a7e4ffcc03b2 not found: ID does not exist" Nov 03 00:06:56.441262 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.441231 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd"] Nov 03 00:06:56.445475 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.445439 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-96031a/distinct-w3-96031a-kserve-6c69b88c98-lmshd"] Nov 03 00:06:56.556576 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.556460 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v"] Nov 03 00:06:56.566061 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.566025 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fr99v"] Nov 03 00:06:56.679988 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.678821 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t"] Nov 03 00:06:56.679988 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.678903 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-96031a/simulator-w3-96031a-kserve-787bf88b74-tmn4t"] Nov 03 00:06:56.793625 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.793577 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="196e5e10-3ef2-4ade-bd72-042921c2f660" path="/var/lib/kubelet/pods/196e5e10-3ef2-4ade-bd72-042921c2f660/volumes" Nov 03 00:06:56.794229 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.794213 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65a85aec-06e9-4952-9a16-055faf8cf19b" path="/var/lib/kubelet/pods/65a85aec-06e9-4952-9a16-055faf8cf19b/volumes" Nov 03 00:06:56.794654 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.794640 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7701f578-a649-4f37-956b-aa1189848710" path="/var/lib/kubelet/pods/7701f578-a649-4f37-956b-aa1189848710/volumes" Nov 03 00:06:56.795362 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.795348 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc58f9ea-5fa3-44d0-975f-1fed35f2463b" path="/var/lib/kubelet/pods/bc58f9ea-5fa3-44d0-975f-1fed35f2463b/volumes" Nov 03 00:06:56.795790 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.795776 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee3fec3e-ad31-4633-b65f-6c97883dea00" path="/var/lib/kubelet/pods/ee3fec3e-ad31-4633-b65f-6c97883dea00/volumes" Nov 03 00:06:56.796136 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.796124 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f850c9fa-e500-4772-8454-a5172e446770" path="/var/lib/kubelet/pods/f850c9fa-e500-4772-8454-a5172e446770/volumes" Nov 03 00:06:56.796796 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:56.796779 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb16fc7d-3def-41dc-91a7-1103ccaf4c35" path="/var/lib/kubelet/pods/fb16fc7d-3def-41dc-91a7-1103ccaf4c35/volumes" Nov 03 00:06:59.784242 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:59.783986 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:06:59.798166 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:06:59.798004 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:07:00.355793 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:00.355761 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:07:07.585518 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:07:07.585444 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:07:26.355127 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.355086 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg"] Nov 03 00:07:26.355677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.355522 5114 cpu_manager.go:414] "containerMap: removing container" podUID="7701f578-a649-4f37-956b-aa1189848710" containerName="istio-proxy" Nov 03 00:07:26.355677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.355540 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7701f578-a649-4f37-956b-aa1189848710" containerName="istio-proxy" Nov 03 00:07:26.355677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.355557 5114 cpu_manager.go:414] "containerMap: removing container" podUID="fb16fc7d-3def-41dc-91a7-1103ccaf4c35" containerName="main" Nov 03 00:07:26.355677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.355562 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb16fc7d-3def-41dc-91a7-1103ccaf4c35" containerName="main" Nov 03 00:07:26.355677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.355571 5114 cpu_manager.go:414] "containerMap: removing container" podUID="65a85aec-06e9-4952-9a16-055faf8cf19b" containerName="istio-proxy" Nov 03 00:07:26.355677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.355576 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65a85aec-06e9-4952-9a16-055faf8cf19b" containerName="istio-proxy" Nov 03 00:07:26.355677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.355585 5114 cpu_manager.go:414] "containerMap: removing container" podUID="ee3fec3e-ad31-4633-b65f-6c97883dea00" containerName="istio-proxy" Nov 03 00:07:26.355677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.355589 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3fec3e-ad31-4633-b65f-6c97883dea00" containerName="istio-proxy" Nov 03 00:07:26.355677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.355596 5114 cpu_manager.go:414] "containerMap: removing container" podUID="a4b5c598-38d9-445d-97a0-724c1a83ef02" containerName="revoke-keys" Nov 03 00:07:26.355677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.355600 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4b5c598-38d9-445d-97a0-724c1a83ef02" containerName="revoke-keys" Nov 03 00:07:26.355677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.355608 5114 cpu_manager.go:414] "containerMap: removing container" podUID="bc58f9ea-5fa3-44d0-975f-1fed35f2463b" containerName="istio-proxy" Nov 03 00:07:26.355677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.355613 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc58f9ea-5fa3-44d0-975f-1fed35f2463b" containerName="istio-proxy" Nov 03 00:07:26.355677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.355624 5114 cpu_manager.go:414] "containerMap: removing container" podUID="196e5e10-3ef2-4ade-bd72-042921c2f660" containerName="istio-proxy" Nov 03 00:07:26.355677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.355628 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="196e5e10-3ef2-4ade-bd72-042921c2f660" containerName="istio-proxy" Nov 03 00:07:26.355677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.355638 5114 cpu_manager.go:414] "containerMap: removing container" podUID="f850c9fa-e500-4772-8454-a5172e446770" containerName="main" Nov 03 00:07:26.355677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.355642 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f850c9fa-e500-4772-8454-a5172e446770" containerName="main" Nov 03 00:07:26.356244 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.355723 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="196e5e10-3ef2-4ade-bd72-042921c2f660" containerName="istio-proxy" Nov 03 00:07:26.356244 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.355733 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="f850c9fa-e500-4772-8454-a5172e446770" containerName="main" Nov 03 00:07:26.356244 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.355743 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="7701f578-a649-4f37-956b-aa1189848710" containerName="istio-proxy" Nov 03 00:07:26.356244 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.355750 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb16fc7d-3def-41dc-91a7-1103ccaf4c35" containerName="main" Nov 03 00:07:26.356244 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.355755 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="65a85aec-06e9-4952-9a16-055faf8cf19b" containerName="istio-proxy" Nov 03 00:07:26.356244 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.355760 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3fec3e-ad31-4633-b65f-6c97883dea00" containerName="istio-proxy" Nov 03 00:07:26.356244 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.355766 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4b5c598-38d9-445d-97a0-724c1a83ef02" containerName="revoke-keys" Nov 03 00:07:26.356244 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.355771 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc58f9ea-5fa3-44d0-975f-1fed35f2463b" containerName="istio-proxy" Nov 03 00:07:26.358866 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.358847 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.410632 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.410600 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-96031a-openshift-default-dockercfg-fx8gh\"" Nov 03 00:07:26.463567 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.463519 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg"] Nov 03 00:07:26.478498 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.478450 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istiod-ca-cert\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.478709 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.478566 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-credential-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.478709 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.478626 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qn5w\" (UniqueName: \"kubernetes.io/projected/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-kube-api-access-6qn5w\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.478709 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.478662 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-data\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.478896 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.478768 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-workload-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.478896 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.478801 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-token\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.478976 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.478890 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-envoy\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.478976 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.478922 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-workload-certs\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.479062 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.478979 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-podinfo\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.580557 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.580521 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istiod-ca-cert\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.580557 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.580563 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-credential-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.580841 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.580594 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qn5w\" (UniqueName: \"kubernetes.io/projected/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-kube-api-access-6qn5w\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.580841 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.580625 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-data\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.580841 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.580658 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-workload-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.580841 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.580678 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-token\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.580841 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.580721 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-envoy\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.580841 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.580744 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-workload-certs\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.580841 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.580796 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-podinfo\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.581225 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.581082 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-credential-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.581225 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.581202 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-data\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.581337 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.581235 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-workload-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.581397 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.581345 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istiod-ca-cert\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.581397 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.581379 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-workload-certs\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.583457 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.583436 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-envoy\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.583645 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.583628 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-podinfo\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.641531 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.641428 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-token\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.641687 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.641571 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qn5w\" (UniqueName: \"kubernetes.io/projected/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-kube-api-access-6qn5w\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:26.672302 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:26.672264 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:27.102701 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:27.102674 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg"] Nov 03 00:07:27.104133 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:07:27.104103 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5daf71a1_5a6a_4edd_8b0f_976da11fea5f.slice/crio-8e4dcda00d69c6085c42b8ccd9e0da28e9a6388689cfd926bb7e8d458bead850 WatchSource:0}: Error finding container 8e4dcda00d69c6085c42b8ccd9e0da28e9a6388689cfd926bb7e8d458bead850: Status 404 returned error can't find the container with id 8e4dcda00d69c6085c42b8ccd9e0da28e9a6388689cfd926bb7e8d458bead850 Nov 03 00:07:27.106619 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:27.106582 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:07:27.106707 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:27.106657 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:07:27.106707 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:27.106684 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:07:27.499188 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:27.499069 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" event={"ID":"5daf71a1-5a6a-4edd-8b0f-976da11fea5f","Type":"ContainerStarted","Data":"c7cb772074bb4e82e65025f4794ae587d1c885b7a80484bd4b3e284192bb8159"} Nov 03 00:07:27.499188 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:27.499126 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" event={"ID":"5daf71a1-5a6a-4edd-8b0f-976da11fea5f","Type":"ContainerStarted","Data":"8e4dcda00d69c6085c42b8ccd9e0da28e9a6388689cfd926bb7e8d458bead850"} Nov 03 00:07:27.628583 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:27.628473 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" podStartSLOduration=1.628453131 podStartE2EDuration="1.628453131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:07:27.624980878 +0000 UTC m=+4722.978014255" watchObservedRunningTime="2026-11-03 00:07:27.628453131 +0000 UTC m=+4722.981486508" Nov 03 00:07:27.673192 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:27.673156 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:28.674529 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:28.674458 5114 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 03 00:07:28.674960 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:28.674547 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" podUID="5daf71a1-5a6a-4edd-8b0f-976da11fea5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 03 00:07:29.673025 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:29.672981 5114 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 03 00:07:29.673232 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:29.673064 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" podUID="5daf71a1-5a6a-4edd-8b0f-976da11fea5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 03 00:07:30.673009 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:30.672968 5114 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 03 00:07:30.673446 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:30.673042 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" podUID="5daf71a1-5a6a-4edd-8b0f-976da11fea5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 03 00:07:31.673393 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:31.673349 5114 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 03 00:07:31.673885 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:31.673426 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" podUID="5daf71a1-5a6a-4edd-8b0f-976da11fea5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 03 00:07:31.791610 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:31.791570 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8"] Nov 03 00:07:31.791826 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:31.791803 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" podUID="03ca2528-0901-4893-9d53-2705cece0aee" containerName="istio-proxy" containerID="cri-o://661b6be539b8f227888754e0ab4727badc6c15256df029bdad084c8514e40e62" gracePeriod=30 Nov 03 00:07:32.672704 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:32.672663 5114 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 03 00:07:32.672939 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:32.672734 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" podUID="5daf71a1-5a6a-4edd-8b0f-976da11fea5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 03 00:07:33.678297 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:33.678263 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:33.678747 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:33.678579 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:33.679345 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:33.679325 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:07:37.052112 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.052079 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:07:37.119565 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.119450 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-workload-socket\") pod \"03ca2528-0901-4893-9d53-2705cece0aee\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " Nov 03 00:07:37.119565 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.119528 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-istio-envoy\") pod \"03ca2528-0901-4893-9d53-2705cece0aee\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " Nov 03 00:07:37.119565 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.119564 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ca2528-0901-4893-9d53-2705cece0aee-istio-token\" (UniqueName: \"kubernetes.io/projected/03ca2528-0901-4893-9d53-2705cece0aee-istio-token\") pod \"03ca2528-0901-4893-9d53-2705cece0aee\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " Nov 03 00:07:37.119868 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.119613 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-credential-socket\") pod \"03ca2528-0901-4893-9d53-2705cece0aee\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " Nov 03 00:07:37.119868 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.119641 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03ca2528-0901-4893-9d53-2705cece0aee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ca2528-0901-4893-9d53-2705cece0aee-istiod-ca-cert\") pod \"03ca2528-0901-4893-9d53-2705cece0aee\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " Nov 03 00:07:37.119868 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.119671 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ca2528-0901-4893-9d53-2705cece0aee-kube-api-access-jsh49\" (UniqueName: \"kubernetes.io/projected/03ca2528-0901-4893-9d53-2705cece0aee-kube-api-access-jsh49\") pod \"03ca2528-0901-4893-9d53-2705cece0aee\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " Nov 03 00:07:37.119868 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.119702 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-workload-certs\") pod \"03ca2528-0901-4893-9d53-2705cece0aee\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " Nov 03 00:07:37.119868 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.119745 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03ca2528-0901-4893-9d53-2705cece0aee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ca2528-0901-4893-9d53-2705cece0aee-istio-podinfo\") pod \"03ca2528-0901-4893-9d53-2705cece0aee\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " Nov 03 00:07:37.119868 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.119766 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-workload-socket" pod "03ca2528-0901-4893-9d53-2705cece0aee" (UID: "03ca2528-0901-4893-9d53-2705cece0aee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:07:37.120157 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.119890 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-credential-socket" pod "03ca2528-0901-4893-9d53-2705cece0aee" (UID: "03ca2528-0901-4893-9d53-2705cece0aee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:07:37.120157 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.119925 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-istio-data\") pod \"03ca2528-0901-4893-9d53-2705cece0aee\" (UID: \"03ca2528-0901-4893-9d53-2705cece0aee\") " Nov 03 00:07:37.120262 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.120241 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-workload-certs" pod "03ca2528-0901-4893-9d53-2705cece0aee" (UID: "03ca2528-0901-4893-9d53-2705cece0aee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:07:37.120321 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.120287 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:37.120321 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.120306 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:37.120440 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.120415 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-istio-data" pod "03ca2528-0901-4893-9d53-2705cece0aee" (UID: "03ca2528-0901-4893-9d53-2705cece0aee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:07:37.120665 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.120637 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ca2528-0901-4893-9d53-2705cece0aee-istiod-ca-cert" pod "03ca2528-0901-4893-9d53-2705cece0aee" (UID: "03ca2528-0901-4893-9d53-2705cece0aee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:07:37.122114 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.122089 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ca2528-0901-4893-9d53-2705cece0aee-istio-token" pod "03ca2528-0901-4893-9d53-2705cece0aee" (UID: "03ca2528-0901-4893-9d53-2705cece0aee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:07:37.122238 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.122091 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-istio-envoy" pod "03ca2528-0901-4893-9d53-2705cece0aee" (UID: "03ca2528-0901-4893-9d53-2705cece0aee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:07:37.122238 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.122188 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03ca2528-0901-4893-9d53-2705cece0aee-istio-podinfo" pod "03ca2528-0901-4893-9d53-2705cece0aee" (UID: "03ca2528-0901-4893-9d53-2705cece0aee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:07:37.122738 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.122719 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ca2528-0901-4893-9d53-2705cece0aee-kube-api-access-jsh49" pod "03ca2528-0901-4893-9d53-2705cece0aee" (UID: "03ca2528-0901-4893-9d53-2705cece0aee"). InnerVolumeSpecName "kube-api-access-jsh49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:07:37.221385 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.221105 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:37.221385 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.221138 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:37.221385 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.221153 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ca2528-0901-4893-9d53-2705cece0aee-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:37.221385 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.221168 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ca2528-0901-4893-9d53-2705cece0aee-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:37.221385 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.221184 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsh49\" (UniqueName: \"kubernetes.io/projected/03ca2528-0901-4893-9d53-2705cece0aee-kube-api-access-jsh49\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:37.221385 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.221198 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ca2528-0901-4893-9d53-2705cece0aee-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:37.221385 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.221211 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ca2528-0901-4893-9d53-2705cece0aee-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:37.584439 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.584396 5114 generic.go:356] "Generic (PLEG): container finished" podID="03ca2528-0901-4893-9d53-2705cece0aee" containerID="661b6be539b8f227888754e0ab4727badc6c15256df029bdad084c8514e40e62" exitCode=0 Nov 03 00:07:37.584658 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.584447 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" event={"ID":"03ca2528-0901-4893-9d53-2705cece0aee","Type":"ContainerDied","Data":"661b6be539b8f227888754e0ab4727badc6c15256df029bdad084c8514e40e62"} Nov 03 00:07:37.584658 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.584459 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" Nov 03 00:07:37.584658 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.584519 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8" event={"ID":"03ca2528-0901-4893-9d53-2705cece0aee","Type":"ContainerDied","Data":"0bb0641e9abadb6dad223b91984b1218569047c70dbf24207813f7cf4b4e1c1a"} Nov 03 00:07:37.584658 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.584543 5114 scope.go:122] "RemoveContainer" containerID="661b6be539b8f227888754e0ab4727badc6c15256df029bdad084c8514e40e62" Nov 03 00:07:37.590370 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:07:37.590332 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:07:37.594903 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.594879 5114 scope.go:122] "RemoveContainer" containerID="f6679ff2142673a7fb4398d1df8e39248230e5ade3549738378dbc984eeb51ed" Nov 03 00:07:37.604653 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.604629 5114 scope.go:122] "RemoveContainer" containerID="661b6be539b8f227888754e0ab4727badc6c15256df029bdad084c8514e40e62" Nov 03 00:07:37.605335 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:07:37.605314 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"661b6be539b8f227888754e0ab4727badc6c15256df029bdad084c8514e40e62\": container with ID starting with 661b6be539b8f227888754e0ab4727badc6c15256df029bdad084c8514e40e62 not found: ID does not exist" containerID="661b6be539b8f227888754e0ab4727badc6c15256df029bdad084c8514e40e62" Nov 03 00:07:37.605444 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.605345 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"661b6be539b8f227888754e0ab4727badc6c15256df029bdad084c8514e40e62"} err="failed to get container status \"661b6be539b8f227888754e0ab4727badc6c15256df029bdad084c8514e40e62\": rpc error: code = NotFound desc = could not find container \"661b6be539b8f227888754e0ab4727badc6c15256df029bdad084c8514e40e62\": container with ID starting with 661b6be539b8f227888754e0ab4727badc6c15256df029bdad084c8514e40e62 not found: ID does not exist" Nov 03 00:07:37.605444 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.605365 5114 scope.go:122] "RemoveContainer" containerID="f6679ff2142673a7fb4398d1df8e39248230e5ade3549738378dbc984eeb51ed" Nov 03 00:07:37.605713 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:07:37.605693 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6679ff2142673a7fb4398d1df8e39248230e5ade3549738378dbc984eeb51ed\": container with ID starting with f6679ff2142673a7fb4398d1df8e39248230e5ade3549738378dbc984eeb51ed not found: ID does not exist" containerID="f6679ff2142673a7fb4398d1df8e39248230e5ade3549738378dbc984eeb51ed" Nov 03 00:07:37.605777 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.605729 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6679ff2142673a7fb4398d1df8e39248230e5ade3549738378dbc984eeb51ed"} err="failed to get container status \"f6679ff2142673a7fb4398d1df8e39248230e5ade3549738378dbc984eeb51ed\": rpc error: code = NotFound desc = could not find container \"f6679ff2142673a7fb4398d1df8e39248230e5ade3549738378dbc984eeb51ed\": container with ID starting with f6679ff2142673a7fb4398d1df8e39248230e5ade3549738378dbc984eeb51ed not found: ID does not exist" Nov 03 00:07:37.748113 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.748076 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8"] Nov 03 00:07:37.798948 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:37.798905 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-04713b77-openshift-default-866b79f589-ljbr8"] Nov 03 00:07:38.793211 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:38.793179 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03ca2528-0901-4893-9d53-2705cece0aee" path="/var/lib/kubelet/pods/03ca2528-0901-4893-9d53-2705cece0aee/volumes" Nov 03 00:07:46.855194 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:46.855157 5114 scope.go:122] "RemoveContainer" containerID="c9514fc2597ae010fd6d968db294103d7736e8352a8e68b05a7b8a06c897a3c8" Nov 03 00:07:48.453931 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:48.453893 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv"] Nov 03 00:07:48.454408 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:48.454107 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" podUID="171d9e93-d16f-4da2-beb7-4526082dbc3f" containerName="maas-api" containerID="cri-o://e1b6bde4d88e0d4da33e0a9b2f0bdf9e4a67e1915f93e84f94ff631a10e4c6a4" gracePeriod=30 Nov 03 00:07:48.656940 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:48.656895 5114 generic.go:356] "Generic (PLEG): container finished" podID="171d9e93-d16f-4da2-beb7-4526082dbc3f" containerID="e1b6bde4d88e0d4da33e0a9b2f0bdf9e4a67e1915f93e84f94ff631a10e4c6a4" exitCode=0 Nov 03 00:07:48.657150 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:48.656981 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" event={"ID":"171d9e93-d16f-4da2-beb7-4526082dbc3f","Type":"ContainerDied","Data":"e1b6bde4d88e0d4da33e0a9b2f0bdf9e4a67e1915f93e84f94ff631a10e4c6a4"} Nov 03 00:07:48.719083 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:48.719010 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" Nov 03 00:07:48.848535 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:48.848464 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/171d9e93-d16f-4da2-beb7-4526082dbc3f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/171d9e93-d16f-4da2-beb7-4526082dbc3f-maas-api-tls\") pod \"171d9e93-d16f-4da2-beb7-4526082dbc3f\" (UID: \"171d9e93-d16f-4da2-beb7-4526082dbc3f\") " Nov 03 00:07:48.848865 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:48.848839 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/171d9e93-d16f-4da2-beb7-4526082dbc3f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/171d9e93-d16f-4da2-beb7-4526082dbc3f-openshift-service-ca\") pod \"171d9e93-d16f-4da2-beb7-4526082dbc3f\" (UID: \"171d9e93-d16f-4da2-beb7-4526082dbc3f\") " Nov 03 00:07:48.849012 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:48.848880 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/171d9e93-d16f-4da2-beb7-4526082dbc3f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/171d9e93-d16f-4da2-beb7-4526082dbc3f-metrics-certs\") pod \"171d9e93-d16f-4da2-beb7-4526082dbc3f\" (UID: \"171d9e93-d16f-4da2-beb7-4526082dbc3f\") " Nov 03 00:07:48.849012 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:48.848973 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/171d9e93-d16f-4da2-beb7-4526082dbc3f-kube-api-access-47nmp\" (UniqueName: \"kubernetes.io/projected/171d9e93-d16f-4da2-beb7-4526082dbc3f-kube-api-access-47nmp\") pod \"171d9e93-d16f-4da2-beb7-4526082dbc3f\" (UID: \"171d9e93-d16f-4da2-beb7-4526082dbc3f\") " Nov 03 00:07:48.850615 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:48.850575 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/171d9e93-d16f-4da2-beb7-4526082dbc3f-openshift-service-ca" pod "171d9e93-d16f-4da2-beb7-4526082dbc3f" (UID: "171d9e93-d16f-4da2-beb7-4526082dbc3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:07:48.852341 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:48.852300 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/171d9e93-d16f-4da2-beb7-4526082dbc3f-metrics-certs" pod "171d9e93-d16f-4da2-beb7-4526082dbc3f" (UID: "171d9e93-d16f-4da2-beb7-4526082dbc3f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:07:48.852540 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:48.852376 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171d9e93-d16f-4da2-beb7-4526082dbc3f-kube-api-access-47nmp" pod "171d9e93-d16f-4da2-beb7-4526082dbc3f" (UID: "171d9e93-d16f-4da2-beb7-4526082dbc3f"). InnerVolumeSpecName "kube-api-access-47nmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:07:48.853061 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:48.853033 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/171d9e93-d16f-4da2-beb7-4526082dbc3f-maas-api-tls" pod "171d9e93-d16f-4da2-beb7-4526082dbc3f" (UID: "171d9e93-d16f-4da2-beb7-4526082dbc3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:07:48.950143 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:48.950108 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/171d9e93-d16f-4da2-beb7-4526082dbc3f-maas-api-tls\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:48.950143 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:48.950142 5114 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/171d9e93-d16f-4da2-beb7-4526082dbc3f-openshift-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:48.950337 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:48.950152 5114 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/171d9e93-d16f-4da2-beb7-4526082dbc3f-metrics-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:48.950337 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:48.950165 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47nmp\" (UniqueName: \"kubernetes.io/projected/171d9e93-d16f-4da2-beb7-4526082dbc3f-kube-api-access-47nmp\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:49.663294 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:49.663259 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" event={"ID":"171d9e93-d16f-4da2-beb7-4526082dbc3f","Type":"ContainerDied","Data":"01bf769b54084c3a91ba256805969c82bd8835daeb35daa31d7e29df0769f6ba"} Nov 03 00:07:49.663294 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:49.663289 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv" Nov 03 00:07:49.663830 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:49.663322 5114 scope.go:122] "RemoveContainer" containerID="e1b6bde4d88e0d4da33e0a9b2f0bdf9e4a67e1915f93e84f94ff631a10e4c6a4" Nov 03 00:07:49.827240 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:49.827203 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv"] Nov 03 00:07:49.830947 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:49.830917 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-96031a-689c96884b-2pfnv"] Nov 03 00:07:50.793050 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:50.793011 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="171d9e93-d16f-4da2-beb7-4526082dbc3f" path="/var/lib/kubelet/pods/171d9e93-d16f-4da2-beb7-4526082dbc3f/volumes" Nov 03 00:07:58.948172 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:58.948130 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg"] Nov 03 00:07:58.948824 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:07:58.948431 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" podUID="5daf71a1-5a6a-4edd-8b0f-976da11fea5f" containerName="istio-proxy" containerID="cri-o://c7cb772074bb4e82e65025f4794ae587d1c885b7a80484bd4b3e284192bb8159" gracePeriod=30 Nov 03 00:08:03.681546 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:03.681500 5114 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:08:03.682018 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:03.681579 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" podUID="5daf71a1-5a6a-4edd-8b0f-976da11fea5f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:08:04.211324 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.211294 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:08:04.309536 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.309496 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-kube-api-access-6qn5w\" (UniqueName: \"kubernetes.io/projected/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-kube-api-access-6qn5w\") pod \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " Nov 03 00:08:04.309536 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.309541 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-envoy\") pod \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " Nov 03 00:08:04.309807 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.309591 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-token\" (UniqueName: \"kubernetes.io/projected/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-token\") pod \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " Nov 03 00:08:04.309807 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.309639 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-data\") pod \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " Nov 03 00:08:04.309807 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.309664 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-workload-socket\") pod \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " Nov 03 00:08:04.309807 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.309704 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-credential-socket\") pod \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " Nov 03 00:08:04.309807 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.309772 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-podinfo\") pod \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " Nov 03 00:08:04.309807 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.309798 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-workload-certs\") pod \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " Nov 03 00:08:04.310145 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.309889 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istiod-ca-cert\") pod \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\" (UID: \"5daf71a1-5a6a-4edd-8b0f-976da11fea5f\") " Nov 03 00:08:04.310145 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.309958 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-workload-socket" pod "5daf71a1-5a6a-4edd-8b0f-976da11fea5f" (UID: "5daf71a1-5a6a-4edd-8b0f-976da11fea5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:04.310145 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.309986 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-credential-socket" pod "5daf71a1-5a6a-4edd-8b0f-976da11fea5f" (UID: "5daf71a1-5a6a-4edd-8b0f-976da11fea5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:04.310145 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.309997 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-data" pod "5daf71a1-5a6a-4edd-8b0f-976da11fea5f" (UID: "5daf71a1-5a6a-4edd-8b0f-976da11fea5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:04.310328 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.310147 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-workload-certs" pod "5daf71a1-5a6a-4edd-8b0f-976da11fea5f" (UID: "5daf71a1-5a6a-4edd-8b0f-976da11fea5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:04.310328 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.310312 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.310398 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.310337 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.310398 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.310353 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.310398 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.310353 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istiod-ca-cert" pod "5daf71a1-5a6a-4edd-8b0f-976da11fea5f" (UID: "5daf71a1-5a6a-4edd-8b0f-976da11fea5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:08:04.310398 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.310368 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.311964 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.311933 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-envoy" pod "5daf71a1-5a6a-4edd-8b0f-976da11fea5f" (UID: "5daf71a1-5a6a-4edd-8b0f-976da11fea5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:04.312158 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.312137 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-token" pod "5daf71a1-5a6a-4edd-8b0f-976da11fea5f" (UID: "5daf71a1-5a6a-4edd-8b0f-976da11fea5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:04.312320 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.312298 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-podinfo" pod "5daf71a1-5a6a-4edd-8b0f-976da11fea5f" (UID: "5daf71a1-5a6a-4edd-8b0f-976da11fea5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:08:04.312422 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.312366 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-kube-api-access-6qn5w" pod "5daf71a1-5a6a-4edd-8b0f-976da11fea5f" (UID: "5daf71a1-5a6a-4edd-8b0f-976da11fea5f"). InnerVolumeSpecName "kube-api-access-6qn5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:04.411276 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.411242 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.411276 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.411271 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.411276 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.411281 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qn5w\" (UniqueName: \"kubernetes.io/projected/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-kube-api-access-6qn5w\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.411663 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.411293 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.411663 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.411303 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5daf71a1-5a6a-4edd-8b0f-976da11fea5f-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.744983 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.744875 5114 generic.go:356] "Generic (PLEG): container finished" podID="5daf71a1-5a6a-4edd-8b0f-976da11fea5f" containerID="c7cb772074bb4e82e65025f4794ae587d1c885b7a80484bd4b3e284192bb8159" exitCode=0 Nov 03 00:08:04.744983 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.744932 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" event={"ID":"5daf71a1-5a6a-4edd-8b0f-976da11fea5f","Type":"ContainerDied","Data":"c7cb772074bb4e82e65025f4794ae587d1c885b7a80484bd4b3e284192bb8159"} Nov 03 00:08:04.744983 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.744941 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" Nov 03 00:08:04.745643 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.744989 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg" event={"ID":"5daf71a1-5a6a-4edd-8b0f-976da11fea5f","Type":"ContainerDied","Data":"8e4dcda00d69c6085c42b8ccd9e0da28e9a6388689cfd926bb7e8d458bead850"} Nov 03 00:08:04.745643 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.745015 5114 scope.go:122] "RemoveContainer" containerID="c7cb772074bb4e82e65025f4794ae587d1c885b7a80484bd4b3e284192bb8159" Nov 03 00:08:04.755382 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.755362 5114 scope.go:122] "RemoveContainer" containerID="c7cb772074bb4e82e65025f4794ae587d1c885b7a80484bd4b3e284192bb8159" Nov 03 00:08:04.755776 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:08:04.755756 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7cb772074bb4e82e65025f4794ae587d1c885b7a80484bd4b3e284192bb8159\": container with ID starting with c7cb772074bb4e82e65025f4794ae587d1c885b7a80484bd4b3e284192bb8159 not found: ID does not exist" containerID="c7cb772074bb4e82e65025f4794ae587d1c885b7a80484bd4b3e284192bb8159" Nov 03 00:08:04.755878 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.755784 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7cb772074bb4e82e65025f4794ae587d1c885b7a80484bd4b3e284192bb8159"} err="failed to get container status \"c7cb772074bb4e82e65025f4794ae587d1c885b7a80484bd4b3e284192bb8159\": rpc error: code = NotFound desc = could not find container \"c7cb772074bb4e82e65025f4794ae587d1c885b7a80484bd4b3e284192bb8159\": container with ID starting with c7cb772074bb4e82e65025f4794ae587d1c885b7a80484bd4b3e284192bb8159 not found: ID does not exist" Nov 03 00:08:04.911720 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.911679 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg"] Nov 03 00:08:04.917928 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:04.917892 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-r5tsg"] Nov 03 00:08:06.792591 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:06.792554 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5daf71a1-5a6a-4edd-8b0f-976da11fea5f" path="/var/lib/kubelet/pods/5daf71a1-5a6a-4edd-8b0f-976da11fea5f/volumes" Nov 03 00:08:07.585070 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:08:07.585027 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:08:28.602979 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.602896 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx"] Nov 03 00:08:28.603364 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.603287 5114 cpu_manager.go:414] "containerMap: removing container" podUID="171d9e93-d16f-4da2-beb7-4526082dbc3f" containerName="maas-api" Nov 03 00:08:28.603364 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.603298 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="171d9e93-d16f-4da2-beb7-4526082dbc3f" containerName="maas-api" Nov 03 00:08:28.603364 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.603316 5114 cpu_manager.go:414] "containerMap: removing container" podUID="5daf71a1-5a6a-4edd-8b0f-976da11fea5f" containerName="istio-proxy" Nov 03 00:08:28.603364 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.603321 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5daf71a1-5a6a-4edd-8b0f-976da11fea5f" containerName="istio-proxy" Nov 03 00:08:28.603364 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.603331 5114 cpu_manager.go:414] "containerMap: removing container" podUID="03ca2528-0901-4893-9d53-2705cece0aee" containerName="istio-proxy" Nov 03 00:08:28.603364 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.603336 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ca2528-0901-4893-9d53-2705cece0aee" containerName="istio-proxy" Nov 03 00:08:28.603593 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.603407 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ca2528-0901-4893-9d53-2705cece0aee" containerName="istio-proxy" Nov 03 00:08:28.603593 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.603419 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="171d9e93-d16f-4da2-beb7-4526082dbc3f" containerName="maas-api" Nov 03 00:08:28.603593 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.603427 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="5daf71a1-5a6a-4edd-8b0f-976da11fea5f" containerName="istio-proxy" Nov 03 00:08:28.606371 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.606353 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:08:28.708133 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.708097 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-b8d3cb\"/\"default-dockercfg-mhnqz\"" Nov 03 00:08:28.708133 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.708103 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-b8d3cb\"/\"openshift-service-ca.crt\"" Nov 03 00:08:28.708133 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.708137 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-b8d3cb\"/\"kube-root-ca.crt\"" Nov 03 00:08:28.708431 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.708170 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-b8d3cb\"/\"sub-test-model-b8d3cb-kserve-self-signed-certs\"" Nov 03 00:08:28.710253 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.710230 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx"] Nov 03 00:08:28.755363 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.755329 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-model-cache\") pod \"sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx\" (UID: \"f9a3771d-32a1-42fc-a172-007488fa523d\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:08:28.755540 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.755375 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89bfr\" (UniqueName: \"kubernetes.io/projected/f9a3771d-32a1-42fc-a172-007488fa523d-kube-api-access-89bfr\") pod \"sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx\" (UID: \"f9a3771d-32a1-42fc-a172-007488fa523d\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:08:28.755540 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.755466 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-tmp-dir\") pod \"sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx\" (UID: \"f9a3771d-32a1-42fc-a172-007488fa523d\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:08:28.755639 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.755547 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9a3771d-32a1-42fc-a172-007488fa523d-tls-certs\") pod \"sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx\" (UID: \"f9a3771d-32a1-42fc-a172-007488fa523d\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:08:28.755639 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.755614 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-home\") pod \"sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx\" (UID: \"f9a3771d-32a1-42fc-a172-007488fa523d\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:08:28.755727 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.755646 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-dshm\") pod \"sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx\" (UID: \"f9a3771d-32a1-42fc-a172-007488fa523d\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:08:28.857330 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.857227 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-model-cache\") pod \"sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx\" (UID: \"f9a3771d-32a1-42fc-a172-007488fa523d\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:08:28.857330 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.857274 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89bfr\" (UniqueName: \"kubernetes.io/projected/f9a3771d-32a1-42fc-a172-007488fa523d-kube-api-access-89bfr\") pod \"sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx\" (UID: \"f9a3771d-32a1-42fc-a172-007488fa523d\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:08:28.857330 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.857311 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-tmp-dir\") pod \"sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx\" (UID: \"f9a3771d-32a1-42fc-a172-007488fa523d\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:08:28.857330 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.857328 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9a3771d-32a1-42fc-a172-007488fa523d-tls-certs\") pod \"sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx\" (UID: \"f9a3771d-32a1-42fc-a172-007488fa523d\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:08:28.857738 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.857505 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-home\") pod \"sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx\" (UID: \"f9a3771d-32a1-42fc-a172-007488fa523d\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:08:28.857738 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.857556 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-dshm\") pod \"sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx\" (UID: \"f9a3771d-32a1-42fc-a172-007488fa523d\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:08:28.857842 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.857738 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-model-cache\") pod \"sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx\" (UID: \"f9a3771d-32a1-42fc-a172-007488fa523d\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:08:28.857842 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.857823 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-tmp-dir\") pod \"sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx\" (UID: \"f9a3771d-32a1-42fc-a172-007488fa523d\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:08:28.857931 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.857894 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-home\") pod \"sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx\" (UID: \"f9a3771d-32a1-42fc-a172-007488fa523d\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:08:28.860169 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.860136 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-dshm\") pod \"sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx\" (UID: \"f9a3771d-32a1-42fc-a172-007488fa523d\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:08:28.860602 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.860572 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9a3771d-32a1-42fc-a172-007488fa523d-tls-certs\") pod \"sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx\" (UID: \"f9a3771d-32a1-42fc-a172-007488fa523d\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:08:28.918074 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.918043 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89bfr\" (UniqueName: \"kubernetes.io/projected/f9a3771d-32a1-42fc-a172-007488fa523d-kube-api-access-89bfr\") pod \"sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx\" (UID: \"f9a3771d-32a1-42fc-a172-007488fa523d\") " pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:08:28.918260 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:28.918237 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:08:29.296968 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:29.296928 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx"] Nov 03 00:08:29.300310 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:08:29.300280 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9a3771d_32a1_42fc_a172_007488fa523d.slice/crio-003f0357327888462f6f70df660b48c3500d7f712da491a0004116e51ceab894 WatchSource:0}: Error finding container 003f0357327888462f6f70df660b48c3500d7f712da491a0004116e51ceab894: Status 404 returned error can't find the container with id 003f0357327888462f6f70df660b48c3500d7f712da491a0004116e51ceab894 Nov 03 00:08:29.879997 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:29.879960 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" event={"ID":"f9a3771d-32a1-42fc-a172-007488fa523d","Type":"ContainerStarted","Data":"828022ce29a8f0a39c061cd5ccd992966b4579c826354b2a31cf84c70eb82770"} Nov 03 00:08:29.880557 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:29.880010 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" event={"ID":"f9a3771d-32a1-42fc-a172-007488fa523d","Type":"ContainerStarted","Data":"003f0357327888462f6f70df660b48c3500d7f712da491a0004116e51ceab894"} Nov 03 00:08:29.918816 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:29.918771 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:08:29.997503 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:29.997416 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" podStartSLOduration=1.997402291 podStartE2EDuration="1.997402291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:08:29.995082338 +0000 UTC m=+4785.348115713" watchObservedRunningTime="2026-11-03 00:08:29.997402291 +0000 UTC m=+4785.350435707" Nov 03 00:08:36.978648 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:36.978604 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8"] Nov 03 00:08:36.979083 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:36.979015 5114 cpu_manager.go:414] "containerMap: removing container" podUID="03ca2528-0901-4893-9d53-2705cece0aee" containerName="istio-proxy" Nov 03 00:08:36.979083 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:36.979029 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ca2528-0901-4893-9d53-2705cece0aee" containerName="istio-proxy" Nov 03 00:08:36.979164 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:36.979114 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ca2528-0901-4893-9d53-2705cece0aee" containerName="istio-proxy" Nov 03 00:08:36.982393 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:36.982374 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:08:37.084231 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:37.084191 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-9b6c9e\"/\"openshift-service-ca.crt\"" Nov 03 00:08:37.084475 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:37.084348 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-9b6c9e\"/\"rate-test-model-9b6c9e-kserve-self-signed-certs\"" Nov 03 00:08:37.084475 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:37.084380 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-9b6c9e\"/\"kube-root-ca.crt\"" Nov 03 00:08:37.084475 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:37.084411 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-9b6c9e\"/\"default-dockercfg-w69v5\"" Nov 03 00:08:37.086715 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:37.086694 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8"] Nov 03 00:08:37.144528 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:37.144474 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk2dh\" (UniqueName: \"kubernetes.io/projected/82b57f4a-5485-4607-b2e9-1c364f3d4b35-kube-api-access-lk2dh\") pod \"rate-test-model-9b6c9e-kserve-84b9cc877-67jv8\" (UID: \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:08:37.144724 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:37.144612 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82b57f4a-5485-4607-b2e9-1c364f3d4b35-tls-certs\") pod \"rate-test-model-9b6c9e-kserve-84b9cc877-67jv8\" (UID: \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:08:37.144724 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:37.144678 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-model-cache\") pod \"rate-test-model-9b6c9e-kserve-84b9cc877-67jv8\" (UID: \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:08:37.144843 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:37.144743 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-tmp-dir\") pod \"rate-test-model-9b6c9e-kserve-84b9cc877-67jv8\" (UID: \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:08:37.144843 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:37.144786 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-home\") pod \"rate-test-model-9b6c9e-kserve-84b9cc877-67jv8\" (UID: \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:08:37.144843 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:37.144813 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-dshm\") pod \"rate-test-model-9b6c9e-kserve-84b9cc877-67jv8\" (UID: \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:08:37.245546 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:37.245421 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-model-cache\") pod \"rate-test-model-9b6c9e-kserve-84b9cc877-67jv8\" (UID: \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:08:37.245546 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:37.245476 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-tmp-dir\") pod \"rate-test-model-9b6c9e-kserve-84b9cc877-67jv8\" (UID: \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:08:37.245546 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:37.245528 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-home\") pod \"rate-test-model-9b6c9e-kserve-84b9cc877-67jv8\" (UID: \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:08:37.245546 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:37.245547 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-dshm\") pod \"rate-test-model-9b6c9e-kserve-84b9cc877-67jv8\" (UID: \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:08:37.245907 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:37.245577 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk2dh\" (UniqueName: \"kubernetes.io/projected/82b57f4a-5485-4607-b2e9-1c364f3d4b35-kube-api-access-lk2dh\") pod \"rate-test-model-9b6c9e-kserve-84b9cc877-67jv8\" (UID: \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:08:37.245907 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:37.245623 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82b57f4a-5485-4607-b2e9-1c364f3d4b35-tls-certs\") pod \"rate-test-model-9b6c9e-kserve-84b9cc877-67jv8\" (UID: \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:08:37.246031 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:37.245922 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-model-cache\") pod \"rate-test-model-9b6c9e-kserve-84b9cc877-67jv8\" (UID: \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:08:37.246031 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:37.245953 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-home\") pod \"rate-test-model-9b6c9e-kserve-84b9cc877-67jv8\" (UID: \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:08:37.246031 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:37.245993 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-tmp-dir\") pod \"rate-test-model-9b6c9e-kserve-84b9cc877-67jv8\" (UID: \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:08:37.248913 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:37.248880 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-dshm\") pod \"rate-test-model-9b6c9e-kserve-84b9cc877-67jv8\" (UID: \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:08:37.249311 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:37.249285 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82b57f4a-5485-4607-b2e9-1c364f3d4b35-tls-certs\") pod \"rate-test-model-9b6c9e-kserve-84b9cc877-67jv8\" (UID: \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:08:37.322318 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:37.322278 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk2dh\" (UniqueName: \"kubernetes.io/projected/82b57f4a-5485-4607-b2e9-1c364f3d4b35-kube-api-access-lk2dh\") pod \"rate-test-model-9b6c9e-kserve-84b9cc877-67jv8\" (UID: \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:08:37.584378 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:08:37.584337 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:08:37.595456 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:37.595427 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:08:37.971317 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:37.971291 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8"] Nov 03 00:08:37.973650 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:08:37.973619 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82b57f4a_5485_4607_b2e9_1c364f3d4b35.slice/crio-958e48d8b2b40afc5c6ead63356dec3b36fdcba802c0a7a1ae25849e0f120e75 WatchSource:0}: Error finding container 958e48d8b2b40afc5c6ead63356dec3b36fdcba802c0a7a1ae25849e0f120e75: Status 404 returned error can't find the container with id 958e48d8b2b40afc5c6ead63356dec3b36fdcba802c0a7a1ae25849e0f120e75 Nov 03 00:08:38.919245 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:38.919203 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:08:38.931246 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:38.931212 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" event={"ID":"82b57f4a-5485-4607-b2e9-1c364f3d4b35","Type":"ContainerStarted","Data":"37f20a646fd0fb4ff89621b90666c9f2b4b91426bfc4da53403f1d8f3003dcc6"} Nov 03 00:08:38.931454 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:38.931256 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" event={"ID":"82b57f4a-5485-4607-b2e9-1c364f3d4b35","Type":"ContainerStarted","Data":"958e48d8b2b40afc5c6ead63356dec3b36fdcba802c0a7a1ae25849e0f120e75"} Nov 03 00:08:38.933345 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:38.933321 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:08:38.944685 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:38.944654 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:08:39.048996 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:39.048930 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" podStartSLOduration=3.048912328 podStartE2EDuration="3.048912328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:08:39.046748328 +0000 UTC m=+4794.399781705" watchObservedRunningTime="2026-11-03 00:08:39.048912328 +0000 UTC m=+4794.401945702" Nov 03 00:08:39.595894 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:39.595854 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:08:42.814065 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:42.814025 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p"] Nov 03 00:08:42.923092 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:42.923057 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8"] Nov 03 00:08:47.596462 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:47.596420 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:08:47.609432 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:47.609404 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:08:47.815047 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:47.814993 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p" podUID="5d1a1875-76ae-454b-9d4c-ee871f1d6a22" containerName="payload-processing" containerID="cri-o://ab99d54b354592a521dea5ffa6c5f8f3981ca9838427afea04ae8288a2a8ee82" gracePeriod=25 Nov 03 00:08:47.923465 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:47.923351 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8" podUID="f278fc6e-68f2-46d0-b405-76993550b9f9" containerName="payload-pre-processing" containerID="cri-o://176e506aa325ecd25b177a3c338ce3d8c63696b9eef1c1af925e0bc719c910da" gracePeriod=25 Nov 03 00:08:47.995169 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:47.995139 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:08:53.491799 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:53.491760 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8" podUID="f278fc6e-68f2-46d0-b405-76993550b9f9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:08:53.492298 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:08:53.491822 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p" podUID="5d1a1875-76ae-454b-9d4c-ee871f1d6a22" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:09:03.491305 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:03.491264 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p" podUID="5d1a1875-76ae-454b-9d4c-ee871f1d6a22" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:09:03.491895 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:03.491328 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8" podUID="f278fc6e-68f2-46d0-b405-76993550b9f9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:09:07.583133 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:09:07.583094 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:09:08.081947 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.081912 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p" Nov 03 00:09:08.088682 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.088627 5114 generic.go:356] "Generic (PLEG): container finished" podID="5d1a1875-76ae-454b-9d4c-ee871f1d6a22" containerID="ab99d54b354592a521dea5ffa6c5f8f3981ca9838427afea04ae8288a2a8ee82" exitCode=0 Nov 03 00:09:08.088682 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.088671 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p" Nov 03 00:09:08.089021 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.088702 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p" event={"ID":"5d1a1875-76ae-454b-9d4c-ee871f1d6a22","Type":"ContainerDied","Data":"ab99d54b354592a521dea5ffa6c5f8f3981ca9838427afea04ae8288a2a8ee82"} Nov 03 00:09:08.089021 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.088768 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p" event={"ID":"5d1a1875-76ae-454b-9d4c-ee871f1d6a22","Type":"ContainerDied","Data":"3da3fed525819f95e72c4563e113cffee1b948fb4705e6ed0bc8f1561c08ff71"} Nov 03 00:09:08.089021 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.088792 5114 scope.go:122] "RemoveContainer" containerID="ab99d54b354592a521dea5ffa6c5f8f3981ca9838427afea04ae8288a2a8ee82" Nov 03 00:09:08.090784 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.090757 5114 generic.go:356] "Generic (PLEG): container finished" podID="f278fc6e-68f2-46d0-b405-76993550b9f9" containerID="176e506aa325ecd25b177a3c338ce3d8c63696b9eef1c1af925e0bc719c910da" exitCode=0 Nov 03 00:09:08.090922 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.090821 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8" event={"ID":"f278fc6e-68f2-46d0-b405-76993550b9f9","Type":"ContainerDied","Data":"176e506aa325ecd25b177a3c338ce3d8c63696b9eef1c1af925e0bc719c910da"} Nov 03 00:09:08.098637 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.098615 5114 scope.go:122] "RemoveContainer" containerID="ab99d54b354592a521dea5ffa6c5f8f3981ca9838427afea04ae8288a2a8ee82" Nov 03 00:09:08.098988 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:09:08.098962 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab99d54b354592a521dea5ffa6c5f8f3981ca9838427afea04ae8288a2a8ee82\": container with ID starting with ab99d54b354592a521dea5ffa6c5f8f3981ca9838427afea04ae8288a2a8ee82 not found: ID does not exist" containerID="ab99d54b354592a521dea5ffa6c5f8f3981ca9838427afea04ae8288a2a8ee82" Nov 03 00:09:08.099077 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.099005 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab99d54b354592a521dea5ffa6c5f8f3981ca9838427afea04ae8288a2a8ee82"} err="failed to get container status \"ab99d54b354592a521dea5ffa6c5f8f3981ca9838427afea04ae8288a2a8ee82\": rpc error: code = NotFound desc = could not find container \"ab99d54b354592a521dea5ffa6c5f8f3981ca9838427afea04ae8288a2a8ee82\": container with ID starting with ab99d54b354592a521dea5ffa6c5f8f3981ca9838427afea04ae8288a2a8ee82 not found: ID does not exist" Nov 03 00:09:08.166851 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.166822 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d1a1875-76ae-454b-9d4c-ee871f1d6a22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d1a1875-76ae-454b-9d4c-ee871f1d6a22-plugins-config-volume\") pod \"5d1a1875-76ae-454b-9d4c-ee871f1d6a22\" (UID: \"5d1a1875-76ae-454b-9d4c-ee871f1d6a22\") " Nov 03 00:09:08.167025 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.166882 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d1a1875-76ae-454b-9d4c-ee871f1d6a22-kube-api-access-7q24h\" (UniqueName: \"kubernetes.io/projected/5d1a1875-76ae-454b-9d4c-ee871f1d6a22-kube-api-access-7q24h\") pod \"5d1a1875-76ae-454b-9d4c-ee871f1d6a22\" (UID: \"5d1a1875-76ae-454b-9d4c-ee871f1d6a22\") " Nov 03 00:09:08.167282 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.167255 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d1a1875-76ae-454b-9d4c-ee871f1d6a22-plugins-config-volume" pod "5d1a1875-76ae-454b-9d4c-ee871f1d6a22" (UID: "5d1a1875-76ae-454b-9d4c-ee871f1d6a22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:09:08.169126 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.169104 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1a1875-76ae-454b-9d4c-ee871f1d6a22-kube-api-access-7q24h" pod "5d1a1875-76ae-454b-9d4c-ee871f1d6a22" (UID: "5d1a1875-76ae-454b-9d4c-ee871f1d6a22"). InnerVolumeSpecName "kube-api-access-7q24h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:09:08.183014 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.182948 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8" Nov 03 00:09:08.267802 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.267767 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f278fc6e-68f2-46d0-b405-76993550b9f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f278fc6e-68f2-46d0-b405-76993550b9f9-plugins-config-volume\") pod \"f278fc6e-68f2-46d0-b405-76993550b9f9\" (UID: \"f278fc6e-68f2-46d0-b405-76993550b9f9\") " Nov 03 00:09:08.268028 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.267856 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f278fc6e-68f2-46d0-b405-76993550b9f9-kube-api-access-n4w8v\" (UniqueName: \"kubernetes.io/projected/f278fc6e-68f2-46d0-b405-76993550b9f9-kube-api-access-n4w8v\") pod \"f278fc6e-68f2-46d0-b405-76993550b9f9\" (UID: \"f278fc6e-68f2-46d0-b405-76993550b9f9\") " Nov 03 00:09:08.268150 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.268134 5114 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d1a1875-76ae-454b-9d4c-ee871f1d6a22-plugins-config-volume\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:08.268224 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.268150 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q24h\" (UniqueName: \"kubernetes.io/projected/5d1a1875-76ae-454b-9d4c-ee871f1d6a22-kube-api-access-7q24h\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:08.268282 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.268254 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f278fc6e-68f2-46d0-b405-76993550b9f9-plugins-config-volume" pod "f278fc6e-68f2-46d0-b405-76993550b9f9" (UID: "f278fc6e-68f2-46d0-b405-76993550b9f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:09:08.270225 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.270200 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f278fc6e-68f2-46d0-b405-76993550b9f9-kube-api-access-n4w8v" pod "f278fc6e-68f2-46d0-b405-76993550b9f9" (UID: "f278fc6e-68f2-46d0-b405-76993550b9f9"). InnerVolumeSpecName "kube-api-access-n4w8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:09:08.368997 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.368947 5114 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f278fc6e-68f2-46d0-b405-76993550b9f9-plugins-config-volume\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:08.368997 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.368991 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4w8v\" (UniqueName: \"kubernetes.io/projected/f278fc6e-68f2-46d0-b405-76993550b9f9-kube-api-access-n4w8v\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:08.508906 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.508816 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p"] Nov 03 00:09:08.512595 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.512554 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-6c3d8e-5cf5ff4669-bj99p"] Nov 03 00:09:08.792430 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:08.792388 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d1a1875-76ae-454b-9d4c-ee871f1d6a22" path="/var/lib/kubelet/pods/5d1a1875-76ae-454b-9d4c-ee871f1d6a22/volumes" Nov 03 00:09:09.098789 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:09.098703 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8" Nov 03 00:09:09.098970 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:09.098705 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8" event={"ID":"f278fc6e-68f2-46d0-b405-76993550b9f9","Type":"ContainerDied","Data":"547b768514a95d5199cd78198f1829037182326267e5273263a3ef4692125512"} Nov 03 00:09:09.098970 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:09.098853 5114 scope.go:122] "RemoveContainer" containerID="176e506aa325ecd25b177a3c338ce3d8c63696b9eef1c1af925e0bc719c910da" Nov 03 00:09:09.262070 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:09.262017 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8"] Nov 03 00:09:09.320860 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:09.320810 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c3d8e-58dd66964c-7tpl8"] Nov 03 00:09:10.792585 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:10.792548 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f278fc6e-68f2-46d0-b405-76993550b9f9" path="/var/lib/kubelet/pods/f278fc6e-68f2-46d0-b405-76993550b9f9/volumes" Nov 03 00:09:20.054401 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:20.054349 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv"] Nov 03 00:09:20.056978 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:20.054656 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" podUID="dc281e12-676c-41ec-9cc9-ac10fab03e26" containerName="maas-api" containerID="cri-o://7c7d2ba603d7bca06c7454a0fb7c50932b1530a920aca8d9a1d3f46a37f26b68" gracePeriod=30 Nov 03 00:09:20.331922 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:20.331890 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" Nov 03 00:09:20.492714 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:20.492677 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc281e12-676c-41ec-9cc9-ac10fab03e26-kube-api-access-4x82s\" (UniqueName: \"kubernetes.io/projected/dc281e12-676c-41ec-9cc9-ac10fab03e26-kube-api-access-4x82s\") pod \"dc281e12-676c-41ec-9cc9-ac10fab03e26\" (UID: \"dc281e12-676c-41ec-9cc9-ac10fab03e26\") " Nov 03 00:09:20.492941 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:20.492878 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc281e12-676c-41ec-9cc9-ac10fab03e26-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc281e12-676c-41ec-9cc9-ac10fab03e26-openshift-service-ca\") pod \"dc281e12-676c-41ec-9cc9-ac10fab03e26\" (UID: \"dc281e12-676c-41ec-9cc9-ac10fab03e26\") " Nov 03 00:09:20.492941 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:20.492911 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc281e12-676c-41ec-9cc9-ac10fab03e26-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc281e12-676c-41ec-9cc9-ac10fab03e26-maas-api-tls\") pod \"dc281e12-676c-41ec-9cc9-ac10fab03e26\" (UID: \"dc281e12-676c-41ec-9cc9-ac10fab03e26\") " Nov 03 00:09:20.492941 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:20.492932 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc281e12-676c-41ec-9cc9-ac10fab03e26-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc281e12-676c-41ec-9cc9-ac10fab03e26-metrics-certs\") pod \"dc281e12-676c-41ec-9cc9-ac10fab03e26\" (UID: \"dc281e12-676c-41ec-9cc9-ac10fab03e26\") " Nov 03 00:09:20.493299 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:20.493272 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc281e12-676c-41ec-9cc9-ac10fab03e26-openshift-service-ca" pod "dc281e12-676c-41ec-9cc9-ac10fab03e26" (UID: "dc281e12-676c-41ec-9cc9-ac10fab03e26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:09:20.495140 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:20.495109 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc281e12-676c-41ec-9cc9-ac10fab03e26-kube-api-access-4x82s" pod "dc281e12-676c-41ec-9cc9-ac10fab03e26" (UID: "dc281e12-676c-41ec-9cc9-ac10fab03e26"). InnerVolumeSpecName "kube-api-access-4x82s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:09:20.495253 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:20.495161 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc281e12-676c-41ec-9cc9-ac10fab03e26-metrics-certs" pod "dc281e12-676c-41ec-9cc9-ac10fab03e26" (UID: "dc281e12-676c-41ec-9cc9-ac10fab03e26"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:09:20.495253 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:20.495194 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc281e12-676c-41ec-9cc9-ac10fab03e26-maas-api-tls" pod "dc281e12-676c-41ec-9cc9-ac10fab03e26" (UID: "dc281e12-676c-41ec-9cc9-ac10fab03e26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:09:20.594813 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:20.594723 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc281e12-676c-41ec-9cc9-ac10fab03e26-maas-api-tls\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:20.594813 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:20.594770 5114 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc281e12-676c-41ec-9cc9-ac10fab03e26-metrics-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:20.594813 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:20.594785 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x82s\" (UniqueName: \"kubernetes.io/projected/dc281e12-676c-41ec-9cc9-ac10fab03e26-kube-api-access-4x82s\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:20.594813 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:20.594799 5114 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc281e12-676c-41ec-9cc9-ac10fab03e26-openshift-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:21.162926 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:21.162883 5114 generic.go:356] "Generic (PLEG): container finished" podID="dc281e12-676c-41ec-9cc9-ac10fab03e26" containerID="7c7d2ba603d7bca06c7454a0fb7c50932b1530a920aca8d9a1d3f46a37f26b68" exitCode=0 Nov 03 00:09:21.163392 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:21.162954 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" Nov 03 00:09:21.163392 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:21.162965 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" event={"ID":"dc281e12-676c-41ec-9cc9-ac10fab03e26","Type":"ContainerDied","Data":"7c7d2ba603d7bca06c7454a0fb7c50932b1530a920aca8d9a1d3f46a37f26b68"} Nov 03 00:09:21.163392 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:21.163030 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv" event={"ID":"dc281e12-676c-41ec-9cc9-ac10fab03e26","Type":"ContainerDied","Data":"3b5b8815d6071ae7bbd7afc9f2bfa0ba7c23f6916e820fe2e1808b21542e1be8"} Nov 03 00:09:21.163392 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:21.163053 5114 scope.go:122] "RemoveContainer" containerID="7c7d2ba603d7bca06c7454a0fb7c50932b1530a920aca8d9a1d3f46a37f26b68" Nov 03 00:09:21.172891 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:21.172867 5114 scope.go:122] "RemoveContainer" containerID="7c7d2ba603d7bca06c7454a0fb7c50932b1530a920aca8d9a1d3f46a37f26b68" Nov 03 00:09:21.173265 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:09:21.173239 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c7d2ba603d7bca06c7454a0fb7c50932b1530a920aca8d9a1d3f46a37f26b68\": container with ID starting with 7c7d2ba603d7bca06c7454a0fb7c50932b1530a920aca8d9a1d3f46a37f26b68 not found: ID does not exist" containerID="7c7d2ba603d7bca06c7454a0fb7c50932b1530a920aca8d9a1d3f46a37f26b68" Nov 03 00:09:21.173322 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:21.173277 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c7d2ba603d7bca06c7454a0fb7c50932b1530a920aca8d9a1d3f46a37f26b68"} err="failed to get container status \"7c7d2ba603d7bca06c7454a0fb7c50932b1530a920aca8d9a1d3f46a37f26b68\": rpc error: code = NotFound desc = could not find container \"7c7d2ba603d7bca06c7454a0fb7c50932b1530a920aca8d9a1d3f46a37f26b68\": container with ID starting with 7c7d2ba603d7bca06c7454a0fb7c50932b1530a920aca8d9a1d3f46a37f26b68 not found: ID does not exist" Nov 03 00:09:21.325848 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:21.325802 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv"] Nov 03 00:09:21.375426 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:21.375388 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c3d8e-c59c58c46-6gspv"] Nov 03 00:09:22.792475 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:22.792436 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc281e12-676c-41ec-9cc9-ac10fab03e26" path="/var/lib/kubelet/pods/dc281e12-676c-41ec-9cc9-ac10fab03e26/volumes" Nov 03 00:09:30.822067 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:30.822024 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7"] Nov 03 00:09:30.822615 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:30.822274 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" podUID="6f16d749-1be4-4d77-bb08-21bcf16bb724" containerName="istio-proxy" containerID="cri-o://a239b19ed4accc4184f3aab36f7e4b71577052c637e030f2303e5f14dd26a0d0" gracePeriod=30 Nov 03 00:09:31.907853 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:31.907810 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d20fd8ec-dgrx2"] Nov 03 00:09:31.908295 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:31.908217 5114 cpu_manager.go:414] "containerMap: removing container" podUID="dc281e12-676c-41ec-9cc9-ac10fab03e26" containerName="maas-api" Nov 03 00:09:31.908295 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:31.908229 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc281e12-676c-41ec-9cc9-ac10fab03e26" containerName="maas-api" Nov 03 00:09:31.908295 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:31.908250 5114 cpu_manager.go:414] "containerMap: removing container" podUID="5d1a1875-76ae-454b-9d4c-ee871f1d6a22" containerName="payload-processing" Nov 03 00:09:31.908295 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:31.908258 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d1a1875-76ae-454b-9d4c-ee871f1d6a22" containerName="payload-processing" Nov 03 00:09:31.908295 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:31.908289 5114 cpu_manager.go:414] "containerMap: removing container" podUID="f278fc6e-68f2-46d0-b405-76993550b9f9" containerName="payload-pre-processing" Nov 03 00:09:31.908295 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:31.908294 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f278fc6e-68f2-46d0-b405-76993550b9f9" containerName="payload-pre-processing" Nov 03 00:09:31.908587 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:31.908360 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc281e12-676c-41ec-9cc9-ac10fab03e26" containerName="maas-api" Nov 03 00:09:31.908587 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:31.908371 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d1a1875-76ae-454b-9d4c-ee871f1d6a22" containerName="payload-processing" Nov 03 00:09:31.908587 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:31.908379 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="f278fc6e-68f2-46d0-b405-76993550b9f9" containerName="payload-pre-processing" Nov 03 00:09:31.911598 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:31.911577 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20fd8ec-dgrx2" Nov 03 00:09:31.963774 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:31.963740 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:09:31.963966 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:31.963858 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 03 00:09:32.005751 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:32.005717 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk74c\" (UniqueName: \"kubernetes.io/projected/874f6db0-2ff4-44b1-a194-c89239749458-kube-api-access-mk74c\") pod \"maas-api-revoke-sub-d20fd8ec-dgrx2\" (UID: \"874f6db0-2ff4-44b1-a194-c89239749458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20fd8ec-dgrx2" Nov 03 00:09:32.005936 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:32.005885 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874f6db0-2ff4-44b1-a194-c89239749458-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d20fd8ec-dgrx2\" (UID: \"874f6db0-2ff4-44b1-a194-c89239749458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20fd8ec-dgrx2" Nov 03 00:09:32.005936 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:32.005924 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874f6db0-2ff4-44b1-a194-c89239749458-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d20fd8ec-dgrx2\" (UID: \"874f6db0-2ff4-44b1-a194-c89239749458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20fd8ec-dgrx2" Nov 03 00:09:32.010393 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:32.010355 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d20fd8ec-dgrx2"] Nov 03 00:09:32.107902 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:32.107848 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874f6db0-2ff4-44b1-a194-c89239749458-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d20fd8ec-dgrx2\" (UID: \"874f6db0-2ff4-44b1-a194-c89239749458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20fd8ec-dgrx2" Nov 03 00:09:32.107902 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:32.107902 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874f6db0-2ff4-44b1-a194-c89239749458-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d20fd8ec-dgrx2\" (UID: \"874f6db0-2ff4-44b1-a194-c89239749458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20fd8ec-dgrx2" Nov 03 00:09:32.108344 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:32.107985 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk74c\" (UniqueName: \"kubernetes.io/projected/874f6db0-2ff4-44b1-a194-c89239749458-kube-api-access-mk74c\") pod \"maas-api-revoke-sub-d20fd8ec-dgrx2\" (UID: \"874f6db0-2ff4-44b1-a194-c89239749458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20fd8ec-dgrx2" Nov 03 00:09:32.108830 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:32.108801 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874f6db0-2ff4-44b1-a194-c89239749458-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d20fd8ec-dgrx2\" (UID: \"874f6db0-2ff4-44b1-a194-c89239749458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20fd8ec-dgrx2" Nov 03 00:09:32.110961 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:32.110931 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874f6db0-2ff4-44b1-a194-c89239749458-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d20fd8ec-dgrx2\" (UID: \"874f6db0-2ff4-44b1-a194-c89239749458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20fd8ec-dgrx2" Nov 03 00:09:32.169168 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:32.169084 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk74c\" (UniqueName: \"kubernetes.io/projected/874f6db0-2ff4-44b1-a194-c89239749458-kube-api-access-mk74c\") pod \"maas-api-revoke-sub-d20fd8ec-dgrx2\" (UID: \"874f6db0-2ff4-44b1-a194-c89239749458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20fd8ec-dgrx2" Nov 03 00:09:32.224465 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:32.224433 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20fd8ec-dgrx2" Nov 03 00:09:32.610825 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:32.610628 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d20fd8ec-dgrx2"] Nov 03 00:09:32.614154 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:09:32.614120 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874f6db0_2ff4_44b1_a194_c89239749458.slice/crio-550a5b475a79a8b524aabe385a3ca8be5836d6455b661d9fd2b8bdf1c1fd0620 WatchSource:0}: Error finding container 550a5b475a79a8b524aabe385a3ca8be5836d6455b661d9fd2b8bdf1c1fd0620: Status 404 returned error can't find the container with id 550a5b475a79a8b524aabe385a3ca8be5836d6455b661d9fd2b8bdf1c1fd0620 Nov 03 00:09:33.234351 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:33.234238 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20fd8ec-dgrx2" event={"ID":"874f6db0-2ff4-44b1-a194-c89239749458","Type":"ContainerStarted","Data":"203cc0290ee328d82006c6dbddac04f889c4907cd4e96704a984676b9b424c5b"} Nov 03 00:09:33.234351 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:33.234311 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20fd8ec-dgrx2" event={"ID":"874f6db0-2ff4-44b1-a194-c89239749458","Type":"ContainerStarted","Data":"550a5b475a79a8b524aabe385a3ca8be5836d6455b661d9fd2b8bdf1c1fd0620"} Nov 03 00:09:33.351515 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:33.351443 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20fd8ec-dgrx2" podStartSLOduration=2.35142495 podStartE2EDuration="2.35142495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:33.349637083 +0000 UTC m=+4848.702670460" watchObservedRunningTime="2026-11-03 00:09:33.35142495 +0000 UTC m=+4848.704458325" Nov 03 00:09:34.241423 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:34.241385 5114 generic.go:356] "Generic (PLEG): container finished" podID="874f6db0-2ff4-44b1-a194-c89239749458" containerID="203cc0290ee328d82006c6dbddac04f889c4907cd4e96704a984676b9b424c5b" exitCode=0 Nov 03 00:09:34.241965 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:34.241471 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20fd8ec-dgrx2" event={"ID":"874f6db0-2ff4-44b1-a194-c89239749458","Type":"ContainerDied","Data":"203cc0290ee328d82006c6dbddac04f889c4907cd4e96704a984676b9b424c5b"} Nov 03 00:09:35.379427 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:35.379398 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20fd8ec-dgrx2" Nov 03 00:09:35.442226 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:35.442185 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/874f6db0-2ff4-44b1-a194-c89239749458-kube-api-access-mk74c\" (UniqueName: \"kubernetes.io/projected/874f6db0-2ff4-44b1-a194-c89239749458-kube-api-access-mk74c\") pod \"874f6db0-2ff4-44b1-a194-c89239749458\" (UID: \"874f6db0-2ff4-44b1-a194-c89239749458\") " Nov 03 00:09:35.442452 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:35.442267 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/874f6db0-2ff4-44b1-a194-c89239749458-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874f6db0-2ff4-44b1-a194-c89239749458-maas-api-openshift-ca\") pod \"874f6db0-2ff4-44b1-a194-c89239749458\" (UID: \"874f6db0-2ff4-44b1-a194-c89239749458\") " Nov 03 00:09:35.442452 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:35.442371 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/874f6db0-2ff4-44b1-a194-c89239749458-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874f6db0-2ff4-44b1-a194-c89239749458-maas-api-xks-ca\") pod \"874f6db0-2ff4-44b1-a194-c89239749458\" (UID: \"874f6db0-2ff4-44b1-a194-c89239749458\") " Nov 03 00:09:35.442674 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:35.442650 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/874f6db0-2ff4-44b1-a194-c89239749458-maas-api-openshift-ca" pod "874f6db0-2ff4-44b1-a194-c89239749458" (UID: "874f6db0-2ff4-44b1-a194-c89239749458"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:09:35.444581 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:35.444553 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/874f6db0-2ff4-44b1-a194-c89239749458-maas-api-xks-ca" pod "874f6db0-2ff4-44b1-a194-c89239749458" (UID: "874f6db0-2ff4-44b1-a194-c89239749458"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:09:35.444666 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:35.444603 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/874f6db0-2ff4-44b1-a194-c89239749458-kube-api-access-mk74c" pod "874f6db0-2ff4-44b1-a194-c89239749458" (UID: "874f6db0-2ff4-44b1-a194-c89239749458"). InnerVolumeSpecName "kube-api-access-mk74c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:09:35.543705 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:35.543669 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874f6db0-2ff4-44b1-a194-c89239749458-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:35.543705 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:35.543700 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk74c\" (UniqueName: \"kubernetes.io/projected/874f6db0-2ff4-44b1-a194-c89239749458-kube-api-access-mk74c\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:35.543705 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:35.543712 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874f6db0-2ff4-44b1-a194-c89239749458-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:35.591232 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:35.591191 5114 patch_prober.go:29] interesting pod/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:09:35.591444 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:35.591261 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" podUID="6f16d749-1be4-4d77-bb08-21bcf16bb724" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:09:36.067702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.067675 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 03 00:09:36.150423 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.150333 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-credential-socket\") pod \"6f16d749-1be4-4d77-bb08-21bcf16bb724\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " Nov 03 00:09:36.150423 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.150378 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-workload-socket\") pod \"6f16d749-1be4-4d77-bb08-21bcf16bb724\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " Nov 03 00:09:36.150713 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.150448 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-token\" (UniqueName: \"kubernetes.io/projected/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-token\") pod \"6f16d749-1be4-4d77-bb08-21bcf16bb724\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " Nov 03 00:09:36.150713 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.150467 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-workload-certs\") pod \"6f16d749-1be4-4d77-bb08-21bcf16bb724\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " Nov 03 00:09:36.150713 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.150620 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f16d749-1be4-4d77-bb08-21bcf16bb724-kube-api-access-l7sml\" (UniqueName: \"kubernetes.io/projected/6f16d749-1be4-4d77-bb08-21bcf16bb724-kube-api-access-l7sml\") pod \"6f16d749-1be4-4d77-bb08-21bcf16bb724\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " Nov 03 00:09:36.150881 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.150726 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-credential-socket" pod "6f16d749-1be4-4d77-bb08-21bcf16bb724" (UID: "6f16d749-1be4-4d77-bb08-21bcf16bb724"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:36.150881 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.150745 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f16d749-1be4-4d77-bb08-21bcf16bb724-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f16d749-1be4-4d77-bb08-21bcf16bb724-istiod-ca-cert\") pod \"6f16d749-1be4-4d77-bb08-21bcf16bb724\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " Nov 03 00:09:36.150881 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.150742 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-workload-socket" pod "6f16d749-1be4-4d77-bb08-21bcf16bb724" (UID: "6f16d749-1be4-4d77-bb08-21bcf16bb724"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:36.150881 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.150800 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-workload-certs" pod "6f16d749-1be4-4d77-bb08-21bcf16bb724" (UID: "6f16d749-1be4-4d77-bb08-21bcf16bb724"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:36.150881 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.150815 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-podinfo\") pod \"6f16d749-1be4-4d77-bb08-21bcf16bb724\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " Nov 03 00:09:36.150881 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.150871 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-data\") pod \"6f16d749-1be4-4d77-bb08-21bcf16bb724\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " Nov 03 00:09:36.151226 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.150917 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-envoy\") pod \"6f16d749-1be4-4d77-bb08-21bcf16bb724\" (UID: \"6f16d749-1be4-4d77-bb08-21bcf16bb724\") " Nov 03 00:09:36.151226 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.151021 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f16d749-1be4-4d77-bb08-21bcf16bb724-istiod-ca-cert" pod "6f16d749-1be4-4d77-bb08-21bcf16bb724" (UID: "6f16d749-1be4-4d77-bb08-21bcf16bb724"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:09:36.151226 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.151150 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-data" pod "6f16d749-1be4-4d77-bb08-21bcf16bb724" (UID: "6f16d749-1be4-4d77-bb08-21bcf16bb724"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:36.151407 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.151382 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f16d749-1be4-4d77-bb08-21bcf16bb724-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:36.151407 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.151403 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:36.151407 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.151418 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:36.151407 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.151433 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:36.151407 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.151447 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:36.153007 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.152978 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-token" pod "6f16d749-1be4-4d77-bb08-21bcf16bb724" (UID: "6f16d749-1be4-4d77-bb08-21bcf16bb724"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:09:36.153121 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.153025 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-podinfo" pod "6f16d749-1be4-4d77-bb08-21bcf16bb724" (UID: "6f16d749-1be4-4d77-bb08-21bcf16bb724"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:09:36.153121 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.153055 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f16d749-1be4-4d77-bb08-21bcf16bb724-kube-api-access-l7sml" pod "6f16d749-1be4-4d77-bb08-21bcf16bb724" (UID: "6f16d749-1be4-4d77-bb08-21bcf16bb724"). InnerVolumeSpecName "kube-api-access-l7sml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:09:36.153121 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.153076 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-envoy" pod "6f16d749-1be4-4d77-bb08-21bcf16bb724" (UID: "6f16d749-1be4-4d77-bb08-21bcf16bb724"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:36.252211 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.252167 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:36.252211 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.252199 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:36.252211 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.252216 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7sml\" (UniqueName: \"kubernetes.io/projected/6f16d749-1be4-4d77-bb08-21bcf16bb724-kube-api-access-l7sml\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:36.252577 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.252231 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f16d749-1be4-4d77-bb08-21bcf16bb724-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:36.253061 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.253042 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20fd8ec-dgrx2" Nov 03 00:09:36.253174 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.253074 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20fd8ec-dgrx2" event={"ID":"874f6db0-2ff4-44b1-a194-c89239749458","Type":"ContainerDied","Data":"550a5b475a79a8b524aabe385a3ca8be5836d6455b661d9fd2b8bdf1c1fd0620"} Nov 03 00:09:36.253174 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.253141 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="550a5b475a79a8b524aabe385a3ca8be5836d6455b661d9fd2b8bdf1c1fd0620" Nov 03 00:09:36.255148 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.255119 5114 generic.go:356] "Generic (PLEG): container finished" podID="6f16d749-1be4-4d77-bb08-21bcf16bb724" containerID="a239b19ed4accc4184f3aab36f7e4b71577052c637e030f2303e5f14dd26a0d0" exitCode=0 Nov 03 00:09:36.255303 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.255188 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" event={"ID":"6f16d749-1be4-4d77-bb08-21bcf16bb724","Type":"ContainerDied","Data":"a239b19ed4accc4184f3aab36f7e4b71577052c637e030f2303e5f14dd26a0d0"} Nov 03 00:09:36.255303 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.255196 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" Nov 03 00:09:36.255303 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.255228 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7" event={"ID":"6f16d749-1be4-4d77-bb08-21bcf16bb724","Type":"ContainerDied","Data":"d1aba4cde2610319214e1844b4010a3fb4198883b68c2a3e9b63d8d5b095392f"} Nov 03 00:09:36.255303 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.255250 5114 scope.go:122] "RemoveContainer" containerID="a239b19ed4accc4184f3aab36f7e4b71577052c637e030f2303e5f14dd26a0d0" Nov 03 00:09:36.264819 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.264790 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8"] Nov 03 00:09:36.270043 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.269936 5114 scope.go:122] "RemoveContainer" containerID="989320271dcb5483ab9e258b6f1ef2e26f3f2f69393dac19dbc9ffe21ec60ff4" Nov 03 00:09:36.270558 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.270531 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd"] Nov 03 00:09:36.281888 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.281847 5114 scope.go:122] "RemoveContainer" containerID="a239b19ed4accc4184f3aab36f7e4b71577052c637e030f2303e5f14dd26a0d0" Nov 03 00:09:36.282306 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:09:36.282276 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a239b19ed4accc4184f3aab36f7e4b71577052c637e030f2303e5f14dd26a0d0\": container with ID starting with a239b19ed4accc4184f3aab36f7e4b71577052c637e030f2303e5f14dd26a0d0 not found: ID does not exist" containerID="a239b19ed4accc4184f3aab36f7e4b71577052c637e030f2303e5f14dd26a0d0" Nov 03 00:09:36.282389 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.282322 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a239b19ed4accc4184f3aab36f7e4b71577052c637e030f2303e5f14dd26a0d0"} err="failed to get container status \"a239b19ed4accc4184f3aab36f7e4b71577052c637e030f2303e5f14dd26a0d0\": rpc error: code = NotFound desc = could not find container \"a239b19ed4accc4184f3aab36f7e4b71577052c637e030f2303e5f14dd26a0d0\": container with ID starting with a239b19ed4accc4184f3aab36f7e4b71577052c637e030f2303e5f14dd26a0d0 not found: ID does not exist" Nov 03 00:09:36.282389 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.282349 5114 scope.go:122] "RemoveContainer" containerID="989320271dcb5483ab9e258b6f1ef2e26f3f2f69393dac19dbc9ffe21ec60ff4" Nov 03 00:09:36.282760 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:09:36.282731 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"989320271dcb5483ab9e258b6f1ef2e26f3f2f69393dac19dbc9ffe21ec60ff4\": container with ID starting with 989320271dcb5483ab9e258b6f1ef2e26f3f2f69393dac19dbc9ffe21ec60ff4 not found: ID does not exist" containerID="989320271dcb5483ab9e258b6f1ef2e26f3f2f69393dac19dbc9ffe21ec60ff4" Nov 03 00:09:36.282885 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.282769 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"989320271dcb5483ab9e258b6f1ef2e26f3f2f69393dac19dbc9ffe21ec60ff4"} err="failed to get container status \"989320271dcb5483ab9e258b6f1ef2e26f3f2f69393dac19dbc9ffe21ec60ff4\": rpc error: code = NotFound desc = could not find container \"989320271dcb5483ab9e258b6f1ef2e26f3f2f69393dac19dbc9ffe21ec60ff4\": container with ID starting with 989320271dcb5483ab9e258b6f1ef2e26f3f2f69393dac19dbc9ffe21ec60ff4 not found: ID does not exist" Nov 03 00:09:36.288387 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.288360 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd"] Nov 03 00:09:36.656661 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.656624 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7"] Nov 03 00:09:36.661903 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.661873 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-6c3d8e-openshift-default-9d6859f87-dtcn7"] Nov 03 00:09:36.793232 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:36.793193 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f16d749-1be4-4d77-bb08-21bcf16bb724" path="/var/lib/kubelet/pods/6f16d749-1be4-4d77-bb08-21bcf16bb724/volumes" Nov 03 00:09:37.583355 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:09:37.583318 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:09:43.918883 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:43.918843 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-283112ef-cr828"] Nov 03 00:09:43.919320 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:43.919234 5114 cpu_manager.go:414] "containerMap: removing container" podUID="6f16d749-1be4-4d77-bb08-21bcf16bb724" containerName="istio-proxy" Nov 03 00:09:43.919320 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:43.919246 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f16d749-1be4-4d77-bb08-21bcf16bb724" containerName="istio-proxy" Nov 03 00:09:43.919320 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:43.919279 5114 cpu_manager.go:414] "containerMap: removing container" podUID="874f6db0-2ff4-44b1-a194-c89239749458" containerName="revoke-keys" Nov 03 00:09:43.919320 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:43.919285 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="874f6db0-2ff4-44b1-a194-c89239749458" containerName="revoke-keys" Nov 03 00:09:43.919469 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:43.919345 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f16d749-1be4-4d77-bb08-21bcf16bb724" containerName="istio-proxy" Nov 03 00:09:43.919469 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:43.919358 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f16d749-1be4-4d77-bb08-21bcf16bb724" containerName="istio-proxy" Nov 03 00:09:43.919469 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:43.919365 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="874f6db0-2ff4-44b1-a194-c89239749458" containerName="revoke-keys" Nov 03 00:09:43.923916 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:43.923893 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-283112ef-cr828" Nov 03 00:09:43.980168 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:43.980134 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-283112ef-cr828"] Nov 03 00:09:44.026303 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:44.026261 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfjwp\" (UniqueName: \"kubernetes.io/projected/c4ef303d-dc7d-41e7-a795-269968a5f7a2-kube-api-access-dfjwp\") pod \"maas-api-revoke-sub-283112ef-cr828\" (UID: \"c4ef303d-dc7d-41e7-a795-269968a5f7a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-283112ef-cr828" Nov 03 00:09:44.026521 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:44.026317 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4ef303d-dc7d-41e7-a795-269968a5f7a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-283112ef-cr828\" (UID: \"c4ef303d-dc7d-41e7-a795-269968a5f7a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-283112ef-cr828" Nov 03 00:09:44.026521 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:44.026466 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4ef303d-dc7d-41e7-a795-269968a5f7a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-283112ef-cr828\" (UID: \"c4ef303d-dc7d-41e7-a795-269968a5f7a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-283112ef-cr828" Nov 03 00:09:44.026919 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:44.026894 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:09:44.026968 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:44.026947 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 03 00:09:44.127255 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:44.127214 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfjwp\" (UniqueName: \"kubernetes.io/projected/c4ef303d-dc7d-41e7-a795-269968a5f7a2-kube-api-access-dfjwp\") pod \"maas-api-revoke-sub-283112ef-cr828\" (UID: \"c4ef303d-dc7d-41e7-a795-269968a5f7a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-283112ef-cr828" Nov 03 00:09:44.127255 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:44.127260 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4ef303d-dc7d-41e7-a795-269968a5f7a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-283112ef-cr828\" (UID: \"c4ef303d-dc7d-41e7-a795-269968a5f7a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-283112ef-cr828" Nov 03 00:09:44.127541 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:44.127342 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4ef303d-dc7d-41e7-a795-269968a5f7a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-283112ef-cr828\" (UID: \"c4ef303d-dc7d-41e7-a795-269968a5f7a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-283112ef-cr828" Nov 03 00:09:44.128069 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:44.128042 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4ef303d-dc7d-41e7-a795-269968a5f7a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-283112ef-cr828\" (UID: \"c4ef303d-dc7d-41e7-a795-269968a5f7a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-283112ef-cr828" Nov 03 00:09:44.129984 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:44.129954 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4ef303d-dc7d-41e7-a795-269968a5f7a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-283112ef-cr828\" (UID: \"c4ef303d-dc7d-41e7-a795-269968a5f7a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-283112ef-cr828" Nov 03 00:09:44.188239 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:44.188169 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfjwp\" (UniqueName: \"kubernetes.io/projected/c4ef303d-dc7d-41e7-a795-269968a5f7a2-kube-api-access-dfjwp\") pod \"maas-api-revoke-sub-283112ef-cr828\" (UID: \"c4ef303d-dc7d-41e7-a795-269968a5f7a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-283112ef-cr828" Nov 03 00:09:44.237911 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:44.237871 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-283112ef-cr828" Nov 03 00:09:44.623934 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:44.623907 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-283112ef-cr828"] Nov 03 00:09:44.626121 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:09:44.626084 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ef303d_dc7d_41e7_a795_269968a5f7a2.slice/crio-02d674aafdee6379d07c1fcd2bb6b3c14aca3e7268fa6b54987694461c125f44 WatchSource:0}: Error finding container 02d674aafdee6379d07c1fcd2bb6b3c14aca3e7268fa6b54987694461c125f44: Status 404 returned error can't find the container with id 02d674aafdee6379d07c1fcd2bb6b3c14aca3e7268fa6b54987694461c125f44 Nov 03 00:09:45.306622 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:45.306579 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-283112ef-cr828" event={"ID":"c4ef303d-dc7d-41e7-a795-269968a5f7a2","Type":"ContainerStarted","Data":"8e50e39815896cb49799a6c5aa2623ec645d25f17c9d4884dcf46ba7660595c3"} Nov 03 00:09:45.307102 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:45.306648 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-283112ef-cr828" event={"ID":"c4ef303d-dc7d-41e7-a795-269968a5f7a2","Type":"ContainerStarted","Data":"02d674aafdee6379d07c1fcd2bb6b3c14aca3e7268fa6b54987694461c125f44"} Nov 03 00:09:45.422171 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:45.422118 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-283112ef-cr828" podStartSLOduration=2.422104063 podStartE2EDuration="2.422104063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:45.420765569 +0000 UTC m=+4860.773798945" watchObservedRunningTime="2026-11-03 00:09:45.422104063 +0000 UTC m=+4860.775137437" Nov 03 00:09:46.314210 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:46.314166 5114 generic.go:356] "Generic (PLEG): container finished" podID="c4ef303d-dc7d-41e7-a795-269968a5f7a2" containerID="8e50e39815896cb49799a6c5aa2623ec645d25f17c9d4884dcf46ba7660595c3" exitCode=0 Nov 03 00:09:46.314659 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:46.314252 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-283112ef-cr828" event={"ID":"c4ef303d-dc7d-41e7-a795-269968a5f7a2","Type":"ContainerDied","Data":"8e50e39815896cb49799a6c5aa2623ec645d25f17c9d4884dcf46ba7660595c3"} Nov 03 00:09:47.468930 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:47.468897 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-283112ef-cr828" Nov 03 00:09:47.564882 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:47.564839 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4ef303d-dc7d-41e7-a795-269968a5f7a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4ef303d-dc7d-41e7-a795-269968a5f7a2-maas-api-openshift-ca\") pod \"c4ef303d-dc7d-41e7-a795-269968a5f7a2\" (UID: \"c4ef303d-dc7d-41e7-a795-269968a5f7a2\") " Nov 03 00:09:47.565092 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:47.564974 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ef303d-dc7d-41e7-a795-269968a5f7a2-kube-api-access-dfjwp\" (UniqueName: \"kubernetes.io/projected/c4ef303d-dc7d-41e7-a795-269968a5f7a2-kube-api-access-dfjwp\") pod \"c4ef303d-dc7d-41e7-a795-269968a5f7a2\" (UID: \"c4ef303d-dc7d-41e7-a795-269968a5f7a2\") " Nov 03 00:09:47.565092 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:47.565040 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4ef303d-dc7d-41e7-a795-269968a5f7a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4ef303d-dc7d-41e7-a795-269968a5f7a2-maas-api-xks-ca\") pod \"c4ef303d-dc7d-41e7-a795-269968a5f7a2\" (UID: \"c4ef303d-dc7d-41e7-a795-269968a5f7a2\") " Nov 03 00:09:47.565346 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:47.565315 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4ef303d-dc7d-41e7-a795-269968a5f7a2-maas-api-openshift-ca" pod "c4ef303d-dc7d-41e7-a795-269968a5f7a2" (UID: "c4ef303d-dc7d-41e7-a795-269968a5f7a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:09:47.567353 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:47.567321 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4ef303d-dc7d-41e7-a795-269968a5f7a2-maas-api-xks-ca" pod "c4ef303d-dc7d-41e7-a795-269968a5f7a2" (UID: "c4ef303d-dc7d-41e7-a795-269968a5f7a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:09:47.567520 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:47.567495 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ef303d-dc7d-41e7-a795-269968a5f7a2-kube-api-access-dfjwp" pod "c4ef303d-dc7d-41e7-a795-269968a5f7a2" (UID: "c4ef303d-dc7d-41e7-a795-269968a5f7a2"). InnerVolumeSpecName "kube-api-access-dfjwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:09:47.666009 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:47.665964 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfjwp\" (UniqueName: \"kubernetes.io/projected/c4ef303d-dc7d-41e7-a795-269968a5f7a2-kube-api-access-dfjwp\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:47.666009 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:47.666004 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4ef303d-dc7d-41e7-a795-269968a5f7a2-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:47.666009 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:47.666017 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4ef303d-dc7d-41e7-a795-269968a5f7a2-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:48.325202 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:48.325166 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-283112ef-cr828" Nov 03 00:09:48.325401 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:48.325200 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-283112ef-cr828" event={"ID":"c4ef303d-dc7d-41e7-a795-269968a5f7a2","Type":"ContainerDied","Data":"02d674aafdee6379d07c1fcd2bb6b3c14aca3e7268fa6b54987694461c125f44"} Nov 03 00:09:48.325401 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:48.325250 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02d674aafdee6379d07c1fcd2bb6b3c14aca3e7268fa6b54987694461c125f44" Nov 03 00:09:51.287711 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.287652 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" podUID="5e3275e5-f49f-4ca7-8d49-03178a77fee6" containerName="main" containerID="cri-o://04aad5caac6eeac5e11f5618e6c4a241b1d3afb1b178209739a9dda4e3a8e698" gracePeriod=45 Nov 03 00:09:51.292272 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.292239 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" podUID="4d705cd8-8b43-4229-9df0-b0980e7728aa" containerName="main" containerID="cri-o://2b93fcca28e10fac7c49321432d4b01ce25724d8e84b35f560df0d477df0f727" gracePeriod=45 Nov 03 00:09:51.314631 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.314403 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" podUID="e9da41e7-2e4b-4912-9267-3f732f714155" containerName="main" containerID="cri-o://f8db1932a558d1205aa11438a5d67b6a0756db6d8680b5fc33ef1c571456e2bd" gracePeriod=45 Nov 03 00:09:51.589502 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.589457 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 03 00:09:51.593254 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.593229 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 03 00:09:51.606242 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.606216 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 03 00:09:51.705794 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.705758 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d705cd8-8b43-4229-9df0-b0980e7728aa-kube-api-access-pgz4n\" (UniqueName: \"kubernetes.io/projected/4d705cd8-8b43-4229-9df0-b0980e7728aa-kube-api-access-pgz4n\") pod \"4d705cd8-8b43-4229-9df0-b0980e7728aa\" (UID: \"4d705cd8-8b43-4229-9df0-b0980e7728aa\") " Nov 03 00:09:51.706093 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.705818 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-tmp-dir\") pod \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\" (UID: \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\") " Nov 03 00:09:51.706093 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.705841 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d705cd8-8b43-4229-9df0-b0980e7728aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d705cd8-8b43-4229-9df0-b0980e7728aa-tls-certs\") pod \"4d705cd8-8b43-4229-9df0-b0980e7728aa\" (UID: \"4d705cd8-8b43-4229-9df0-b0980e7728aa\") " Nov 03 00:09:51.706093 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.705872 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-tmp-dir\") pod \"e9da41e7-2e4b-4912-9267-3f732f714155\" (UID: \"e9da41e7-2e4b-4912-9267-3f732f714155\") " Nov 03 00:09:51.706093 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.705900 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-model-cache\") pod \"4d705cd8-8b43-4229-9df0-b0980e7728aa\" (UID: \"4d705cd8-8b43-4229-9df0-b0980e7728aa\") " Nov 03 00:09:51.706093 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.705947 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e3275e5-f49f-4ca7-8d49-03178a77fee6-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e3275e5-f49f-4ca7-8d49-03178a77fee6-tls-certs\") pod \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\" (UID: \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\") " Nov 03 00:09:51.706093 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.705996 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-tmp-dir\") pod \"4d705cd8-8b43-4229-9df0-b0980e7728aa\" (UID: \"4d705cd8-8b43-4229-9df0-b0980e7728aa\") " Nov 03 00:09:51.706093 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.706023 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-home\" (UniqueName: \"kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-home\") pod \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\" (UID: \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\") " Nov 03 00:09:51.706093 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.706063 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-home\" (UniqueName: \"kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-home\") pod \"e9da41e7-2e4b-4912-9267-3f732f714155\" (UID: \"e9da41e7-2e4b-4912-9267-3f732f714155\") " Nov 03 00:09:51.706093 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.706093 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-dshm\") pod \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\" (UID: \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\") " Nov 03 00:09:51.706636 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.706117 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-tmp-dir" pod "5e3275e5-f49f-4ca7-8d49-03178a77fee6" (UID: "5e3275e5-f49f-4ca7-8d49-03178a77fee6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:51.706636 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.706124 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-model-cache\") pod \"e9da41e7-2e4b-4912-9267-3f732f714155\" (UID: \"e9da41e7-2e4b-4912-9267-3f732f714155\") " Nov 03 00:09:51.706636 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.706182 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-dshm\") pod \"e9da41e7-2e4b-4912-9267-3f732f714155\" (UID: \"e9da41e7-2e4b-4912-9267-3f732f714155\") " Nov 03 00:09:51.706636 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.706292 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-home\" (UniqueName: \"kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-home\") pod \"4d705cd8-8b43-4229-9df0-b0980e7728aa\" (UID: \"4d705cd8-8b43-4229-9df0-b0980e7728aa\") " Nov 03 00:09:51.706636 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.706304 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-model-cache" pod "e9da41e7-2e4b-4912-9267-3f732f714155" (UID: "e9da41e7-2e4b-4912-9267-3f732f714155"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:51.706636 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.706325 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9da41e7-2e4b-4912-9267-3f732f714155-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9da41e7-2e4b-4912-9267-3f732f714155-tls-certs\") pod \"e9da41e7-2e4b-4912-9267-3f732f714155\" (UID: \"e9da41e7-2e4b-4912-9267-3f732f714155\") " Nov 03 00:09:51.706636 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.706379 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-model-cache\") pod \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\" (UID: \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\") " Nov 03 00:09:51.706636 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.706430 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e3275e5-f49f-4ca7-8d49-03178a77fee6-kube-api-access-mmflk\" (UniqueName: \"kubernetes.io/projected/5e3275e5-f49f-4ca7-8d49-03178a77fee6-kube-api-access-mmflk\") pod \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\" (UID: \"5e3275e5-f49f-4ca7-8d49-03178a77fee6\") " Nov 03 00:09:51.707584 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.707552 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-dshm\") pod \"4d705cd8-8b43-4229-9df0-b0980e7728aa\" (UID: \"4d705cd8-8b43-4229-9df0-b0980e7728aa\") " Nov 03 00:09:51.707736 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.707620 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9da41e7-2e4b-4912-9267-3f732f714155-kube-api-access-wvxzf\" (UniqueName: \"kubernetes.io/projected/e9da41e7-2e4b-4912-9267-3f732f714155-kube-api-access-wvxzf\") pod \"e9da41e7-2e4b-4912-9267-3f732f714155\" (UID: \"e9da41e7-2e4b-4912-9267-3f732f714155\") " Nov 03 00:09:51.708096 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.708047 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:51.708096 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.708075 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:51.708891 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.708643 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-tmp-dir" pod "4d705cd8-8b43-4229-9df0-b0980e7728aa" (UID: "4d705cd8-8b43-4229-9df0-b0980e7728aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:51.709350 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.708942 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-home" pod "e9da41e7-2e4b-4912-9267-3f732f714155" (UID: "e9da41e7-2e4b-4912-9267-3f732f714155"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:51.709350 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.709223 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-home" pod "4d705cd8-8b43-4229-9df0-b0980e7728aa" (UID: "4d705cd8-8b43-4229-9df0-b0980e7728aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:51.709350 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.709283 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-model-cache" pod "5e3275e5-f49f-4ca7-8d49-03178a77fee6" (UID: "5e3275e5-f49f-4ca7-8d49-03178a77fee6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:51.709350 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.709300 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-home" pod "5e3275e5-f49f-4ca7-8d49-03178a77fee6" (UID: "5e3275e5-f49f-4ca7-8d49-03178a77fee6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:51.709655 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.709462 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-tmp-dir" pod "e9da41e7-2e4b-4912-9267-3f732f714155" (UID: "e9da41e7-2e4b-4912-9267-3f732f714155"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:51.709730 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.709676 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-model-cache" pod "4d705cd8-8b43-4229-9df0-b0980e7728aa" (UID: "4d705cd8-8b43-4229-9df0-b0980e7728aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:51.710371 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.710343 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d705cd8-8b43-4229-9df0-b0980e7728aa-tls-certs" pod "4d705cd8-8b43-4229-9df0-b0980e7728aa" (UID: "4d705cd8-8b43-4229-9df0-b0980e7728aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:09:51.710661 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.710577 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-dshm" pod "e9da41e7-2e4b-4912-9267-3f732f714155" (UID: "e9da41e7-2e4b-4912-9267-3f732f714155"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:51.710962 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.710933 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9da41e7-2e4b-4912-9267-3f732f714155-kube-api-access-wvxzf" pod "e9da41e7-2e4b-4912-9267-3f732f714155" (UID: "e9da41e7-2e4b-4912-9267-3f732f714155"). InnerVolumeSpecName "kube-api-access-wvxzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:09:51.711060 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.711006 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d705cd8-8b43-4229-9df0-b0980e7728aa-kube-api-access-pgz4n" pod "4d705cd8-8b43-4229-9df0-b0980e7728aa" (UID: "4d705cd8-8b43-4229-9df0-b0980e7728aa"). InnerVolumeSpecName "kube-api-access-pgz4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:09:51.711717 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.711698 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e3275e5-f49f-4ca7-8d49-03178a77fee6-kube-api-access-mmflk" pod "5e3275e5-f49f-4ca7-8d49-03178a77fee6" (UID: "5e3275e5-f49f-4ca7-8d49-03178a77fee6"). InnerVolumeSpecName "kube-api-access-mmflk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:09:51.712260 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.712230 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-dshm" pod "4d705cd8-8b43-4229-9df0-b0980e7728aa" (UID: "4d705cd8-8b43-4229-9df0-b0980e7728aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:51.712829 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.712799 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-dshm" pod "5e3275e5-f49f-4ca7-8d49-03178a77fee6" (UID: "5e3275e5-f49f-4ca7-8d49-03178a77fee6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:51.712990 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.712970 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9da41e7-2e4b-4912-9267-3f732f714155-tls-certs" pod "e9da41e7-2e4b-4912-9267-3f732f714155" (UID: "e9da41e7-2e4b-4912-9267-3f732f714155"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:09:51.713704 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.713678 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e3275e5-f49f-4ca7-8d49-03178a77fee6-tls-certs" pod "5e3275e5-f49f-4ca7-8d49-03178a77fee6" (UID: "5e3275e5-f49f-4ca7-8d49-03178a77fee6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:09:51.809551 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.809511 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:51.809551 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.809540 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9da41e7-2e4b-4912-9267-3f732f714155-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:51.809551 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.809552 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:51.809804 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.809568 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmflk\" (UniqueName: \"kubernetes.io/projected/5e3275e5-f49f-4ca7-8d49-03178a77fee6-kube-api-access-mmflk\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:51.809804 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.809580 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:51.809804 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.809589 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvxzf\" (UniqueName: \"kubernetes.io/projected/e9da41e7-2e4b-4912-9267-3f732f714155-kube-api-access-wvxzf\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:51.809804 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.809600 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgz4n\" (UniqueName: \"kubernetes.io/projected/4d705cd8-8b43-4229-9df0-b0980e7728aa-kube-api-access-pgz4n\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:51.809804 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.809610 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d705cd8-8b43-4229-9df0-b0980e7728aa-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:51.809804 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.809618 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:51.809804 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.809627 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:51.809804 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.809640 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e3275e5-f49f-4ca7-8d49-03178a77fee6-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:51.809804 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.809648 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d705cd8-8b43-4229-9df0-b0980e7728aa-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:51.809804 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.809656 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:51.809804 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.809664 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:51.809804 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.809670 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e3275e5-f49f-4ca7-8d49-03178a77fee6-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:51.809804 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:51.809678 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9da41e7-2e4b-4912-9267-3f732f714155-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:52.348603 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.348563 5114 generic.go:356] "Generic (PLEG): container finished" podID="5e3275e5-f49f-4ca7-8d49-03178a77fee6" containerID="04aad5caac6eeac5e11f5618e6c4a241b1d3afb1b178209739a9dda4e3a8e698" exitCode=0 Nov 03 00:09:52.349153 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.348611 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" Nov 03 00:09:52.349153 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.348646 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" event={"ID":"5e3275e5-f49f-4ca7-8d49-03178a77fee6","Type":"ContainerDied","Data":"04aad5caac6eeac5e11f5618e6c4a241b1d3afb1b178209739a9dda4e3a8e698"} Nov 03 00:09:52.349153 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.348692 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8" event={"ID":"5e3275e5-f49f-4ca7-8d49-03178a77fee6","Type":"ContainerDied","Data":"2ffeaf44ef293fd5bc05153ebaced2644a9ff36d4ff255154b9ea5242d2cb11f"} Nov 03 00:09:52.349153 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.348715 5114 scope.go:122] "RemoveContainer" containerID="04aad5caac6eeac5e11f5618e6c4a241b1d3afb1b178209739a9dda4e3a8e698" Nov 03 00:09:52.350325 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.350300 5114 generic.go:356] "Generic (PLEG): container finished" podID="e9da41e7-2e4b-4912-9267-3f732f714155" containerID="f8db1932a558d1205aa11438a5d67b6a0756db6d8680b5fc33ef1c571456e2bd" exitCode=0 Nov 03 00:09:52.350434 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.350358 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" Nov 03 00:09:52.350434 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.350355 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" event={"ID":"e9da41e7-2e4b-4912-9267-3f732f714155","Type":"ContainerDied","Data":"f8db1932a558d1205aa11438a5d67b6a0756db6d8680b5fc33ef1c571456e2bd"} Nov 03 00:09:52.350434 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.350425 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd" event={"ID":"e9da41e7-2e4b-4912-9267-3f732f714155","Type":"ContainerDied","Data":"daf2d4e34c9bf34e7b3700d64000d0de535af3503356357a4411eb1cc26a10d5"} Nov 03 00:09:52.356514 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.351900 5114 generic.go:356] "Generic (PLEG): container finished" podID="4d705cd8-8b43-4229-9df0-b0980e7728aa" containerID="2b93fcca28e10fac7c49321432d4b01ce25724d8e84b35f560df0d477df0f727" exitCode=0 Nov 03 00:09:52.356514 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.351989 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" event={"ID":"4d705cd8-8b43-4229-9df0-b0980e7728aa","Type":"ContainerDied","Data":"2b93fcca28e10fac7c49321432d4b01ce25724d8e84b35f560df0d477df0f727"} Nov 03 00:09:52.356514 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.352013 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" event={"ID":"4d705cd8-8b43-4229-9df0-b0980e7728aa","Type":"ContainerDied","Data":"7308c25b56562a8b9d25c2e1884d5a408242f90ca71e22de8db7fbf2c934b6ba"} Nov 03 00:09:52.356514 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.352059 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd" Nov 03 00:09:52.360709 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.360651 5114 scope.go:122] "RemoveContainer" containerID="04aad5caac6eeac5e11f5618e6c4a241b1d3afb1b178209739a9dda4e3a8e698" Nov 03 00:09:52.361150 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:09:52.361120 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04aad5caac6eeac5e11f5618e6c4a241b1d3afb1b178209739a9dda4e3a8e698\": container with ID starting with 04aad5caac6eeac5e11f5618e6c4a241b1d3afb1b178209739a9dda4e3a8e698 not found: ID does not exist" containerID="04aad5caac6eeac5e11f5618e6c4a241b1d3afb1b178209739a9dda4e3a8e698" Nov 03 00:09:52.361250 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.361164 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04aad5caac6eeac5e11f5618e6c4a241b1d3afb1b178209739a9dda4e3a8e698"} err="failed to get container status \"04aad5caac6eeac5e11f5618e6c4a241b1d3afb1b178209739a9dda4e3a8e698\": rpc error: code = NotFound desc = could not find container \"04aad5caac6eeac5e11f5618e6c4a241b1d3afb1b178209739a9dda4e3a8e698\": container with ID starting with 04aad5caac6eeac5e11f5618e6c4a241b1d3afb1b178209739a9dda4e3a8e698 not found: ID does not exist" Nov 03 00:09:52.361250 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.361189 5114 scope.go:122] "RemoveContainer" containerID="f8db1932a558d1205aa11438a5d67b6a0756db6d8680b5fc33ef1c571456e2bd" Nov 03 00:09:52.370217 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.370195 5114 scope.go:122] "RemoveContainer" containerID="f8db1932a558d1205aa11438a5d67b6a0756db6d8680b5fc33ef1c571456e2bd" Nov 03 00:09:52.370591 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:09:52.370569 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8db1932a558d1205aa11438a5d67b6a0756db6d8680b5fc33ef1c571456e2bd\": container with ID starting with f8db1932a558d1205aa11438a5d67b6a0756db6d8680b5fc33ef1c571456e2bd not found: ID does not exist" containerID="f8db1932a558d1205aa11438a5d67b6a0756db6d8680b5fc33ef1c571456e2bd" Nov 03 00:09:52.370657 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.370602 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8db1932a558d1205aa11438a5d67b6a0756db6d8680b5fc33ef1c571456e2bd"} err="failed to get container status \"f8db1932a558d1205aa11438a5d67b6a0756db6d8680b5fc33ef1c571456e2bd\": rpc error: code = NotFound desc = could not find container \"f8db1932a558d1205aa11438a5d67b6a0756db6d8680b5fc33ef1c571456e2bd\": container with ID starting with f8db1932a558d1205aa11438a5d67b6a0756db6d8680b5fc33ef1c571456e2bd not found: ID does not exist" Nov 03 00:09:52.370657 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.370625 5114 scope.go:122] "RemoveContainer" containerID="2b93fcca28e10fac7c49321432d4b01ce25724d8e84b35f560df0d477df0f727" Nov 03 00:09:52.379887 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.379863 5114 scope.go:122] "RemoveContainer" containerID="2b93fcca28e10fac7c49321432d4b01ce25724d8e84b35f560df0d477df0f727" Nov 03 00:09:52.380250 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:09:52.380231 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b93fcca28e10fac7c49321432d4b01ce25724d8e84b35f560df0d477df0f727\": container with ID starting with 2b93fcca28e10fac7c49321432d4b01ce25724d8e84b35f560df0d477df0f727 not found: ID does not exist" containerID="2b93fcca28e10fac7c49321432d4b01ce25724d8e84b35f560df0d477df0f727" Nov 03 00:09:52.380290 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.380261 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b93fcca28e10fac7c49321432d4b01ce25724d8e84b35f560df0d477df0f727"} err="failed to get container status \"2b93fcca28e10fac7c49321432d4b01ce25724d8e84b35f560df0d477df0f727\": rpc error: code = NotFound desc = could not find container \"2b93fcca28e10fac7c49321432d4b01ce25724d8e84b35f560df0d477df0f727\": container with ID starting with 2b93fcca28e10fac7c49321432d4b01ce25724d8e84b35f560df0d477df0f727 not found: ID does not exist" Nov 03 00:09:52.511672 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.511633 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8"] Nov 03 00:09:52.561285 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.561196 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-637fdb/distinct-2-w4-637fdb-kserve-5d86bcc8bb-h6nz8"] Nov 03 00:09:52.669280 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.669234 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd"] Nov 03 00:09:52.673573 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.673537 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-637fdb/simulator-w4-637fdb-kserve-566fc5cc95-xn8nd"] Nov 03 00:09:52.781690 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.781648 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd"] Nov 03 00:09:52.785401 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.785367 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-637fdb/distinct-w4-637fdb-kserve-64dfd6dfdf-kbscd"] Nov 03 00:09:52.792810 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.792779 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d705cd8-8b43-4229-9df0-b0980e7728aa" path="/var/lib/kubelet/pods/4d705cd8-8b43-4229-9df0-b0980e7728aa/volumes" Nov 03 00:09:52.793299 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.793279 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e3275e5-f49f-4ca7-8d49-03178a77fee6" path="/var/lib/kubelet/pods/5e3275e5-f49f-4ca7-8d49-03178a77fee6/volumes" Nov 03 00:09:52.793745 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:09:52.793727 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9da41e7-2e4b-4912-9267-3f732f714155" path="/var/lib/kubelet/pods/e9da41e7-2e4b-4912-9267-3f732f714155/volumes" Nov 03 00:10:07.583959 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:10:07.583919 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:10:13.632643 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.632604 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm"] Nov 03 00:10:13.633125 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.633110 5114 cpu_manager.go:414] "containerMap: removing container" podUID="5e3275e5-f49f-4ca7-8d49-03178a77fee6" containerName="main" Nov 03 00:10:13.633168 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.633128 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e3275e5-f49f-4ca7-8d49-03178a77fee6" containerName="main" Nov 03 00:10:13.633168 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.633158 5114 cpu_manager.go:414] "containerMap: removing container" podUID="c4ef303d-dc7d-41e7-a795-269968a5f7a2" containerName="revoke-keys" Nov 03 00:10:13.633240 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.633165 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ef303d-dc7d-41e7-a795-269968a5f7a2" containerName="revoke-keys" Nov 03 00:10:13.633240 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.633181 5114 cpu_manager.go:414] "containerMap: removing container" podUID="4d705cd8-8b43-4229-9df0-b0980e7728aa" containerName="main" Nov 03 00:10:13.633240 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.633190 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d705cd8-8b43-4229-9df0-b0980e7728aa" containerName="main" Nov 03 00:10:13.633240 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.633205 5114 cpu_manager.go:414] "containerMap: removing container" podUID="6f16d749-1be4-4d77-bb08-21bcf16bb724" containerName="istio-proxy" Nov 03 00:10:13.633240 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.633214 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f16d749-1be4-4d77-bb08-21bcf16bb724" containerName="istio-proxy" Nov 03 00:10:13.633240 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.633230 5114 cpu_manager.go:414] "containerMap: removing container" podUID="e9da41e7-2e4b-4912-9267-3f732f714155" containerName="main" Nov 03 00:10:13.633240 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.633237 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9da41e7-2e4b-4912-9267-3f732f714155" containerName="main" Nov 03 00:10:13.633460 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.633312 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e3275e5-f49f-4ca7-8d49-03178a77fee6" containerName="main" Nov 03 00:10:13.633460 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.633325 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4ef303d-dc7d-41e7-a795-269968a5f7a2" containerName="revoke-keys" Nov 03 00:10:13.633460 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.633333 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d705cd8-8b43-4229-9df0-b0980e7728aa" containerName="main" Nov 03 00:10:13.633460 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.633343 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9da41e7-2e4b-4912-9267-3f732f714155" containerName="main" Nov 03 00:10:13.636784 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.636758 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.688805 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.688772 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-637fdb-openshift-default-dockercfg-brzsm\"" Nov 03 00:10:13.712404 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.712351 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76sc7\" (UniqueName: \"kubernetes.io/projected/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-kube-api-access-76sc7\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.712404 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.712409 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.712677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.712521 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.712677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.712550 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.712677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.712590 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.712677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.712645 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.712815 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.712731 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.712815 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.712751 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.712815 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.712774 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.740915 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.740880 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm"] Nov 03 00:10:13.813846 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.813800 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.814076 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.813892 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.814076 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.813918 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.814076 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.813975 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.814076 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.814003 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.814076 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.814065 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.814366 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.814090 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.814366 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.814119 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.814366 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.814190 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76sc7\" (UniqueName: \"kubernetes.io/projected/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-kube-api-access-76sc7\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.814366 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.814291 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.814366 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.814333 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.814686 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.814622 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.814686 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.814655 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.814844 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.814807 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.816814 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.816781 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.817278 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.817255 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.875266 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.875227 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.875466 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.875368 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76sc7\" (UniqueName: \"kubernetes.io/projected/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-kube-api-access-76sc7\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:13.950912 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:13.950804 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:14.388185 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:14.388153 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm"] Nov 03 00:10:14.390007 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:10:14.389975 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3aa3e9_705b_4a34_a4af_b964c6991e7c.slice/crio-2af48f25bb09848fc78e6a82c7a9f5d89124d73a3bba4caca2a85134188dcefa WatchSource:0}: Error finding container 2af48f25bb09848fc78e6a82c7a9f5d89124d73a3bba4caca2a85134188dcefa: Status 404 returned error can't find the container with id 2af48f25bb09848fc78e6a82c7a9f5d89124d73a3bba4caca2a85134188dcefa Nov 03 00:10:14.392512 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:14.392445 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:10:14.393043 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:14.392561 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:10:14.393043 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:14.392595 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:10:14.480135 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:14.480098 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" event={"ID":"fd3aa3e9-705b-4a34-a4af-b964c6991e7c","Type":"ContainerStarted","Data":"c7a3013c6eb6abf99fd3c7d8fbb88d6945dbef1dbbfc6fa607b97a9c13bf990f"} Nov 03 00:10:14.480312 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:14.480164 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" event={"ID":"fd3aa3e9-705b-4a34-a4af-b964c6991e7c","Type":"ContainerStarted","Data":"2af48f25bb09848fc78e6a82c7a9f5d89124d73a3bba4caca2a85134188dcefa"} Nov 03 00:10:14.599655 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:14.599582 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" podStartSLOduration=1.5995613880000001 podStartE2EDuration="1.599561388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:14.598936568 +0000 UTC m=+4889.951969944" watchObservedRunningTime="2026-11-03 00:10:14.599561388 +0000 UTC m=+4889.952594764" Nov 03 00:10:14.951496 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:14.951446 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:15.952837 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:15.952797 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 03 00:10:15.953307 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:15.952857 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" podUID="fd3aa3e9-705b-4a34-a4af-b964c6991e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 03 00:10:16.952039 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:16.952003 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 03 00:10:16.952245 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:16.952059 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" podUID="fd3aa3e9-705b-4a34-a4af-b964c6991e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 03 00:10:17.951582 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:17.951539 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 03 00:10:17.952051 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:17.951601 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" podUID="fd3aa3e9-705b-4a34-a4af-b964c6991e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 03 00:10:18.951222 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:18.951186 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 03 00:10:18.951423 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:18.951238 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" podUID="fd3aa3e9-705b-4a34-a4af-b964c6991e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 03 00:10:19.953213 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:19.953167 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 03 00:10:19.953645 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:19.953228 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" podUID="fd3aa3e9-705b-4a34-a4af-b964c6991e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 03 00:10:20.951761 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:20.951720 5114 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 03 00:10:20.951965 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:20.951786 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" podUID="fd3aa3e9-705b-4a34-a4af-b964c6991e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 03 00:10:21.956306 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:21.956268 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:21.956853 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:21.956589 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:21.957432 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:21.957414 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:34.082405 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:34.082363 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-726e8bf6-dqtdj"] Nov 03 00:10:34.094521 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:34.094458 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-726e8bf6-dqtdj"] Nov 03 00:10:34.792657 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:34.792610 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f78f19e-8722-4269-919c-289f0da197fa" path="/var/lib/kubelet/pods/9f78f19e-8722-4269-919c-289f0da197fa/volumes" Nov 03 00:10:36.045612 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:36.045577 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f039fc5-lc96x"] Nov 03 00:10:36.095293 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:36.095249 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f039fc5-lc96x"] Nov 03 00:10:36.794621 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:36.794585 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbb7a506-9811-4530-bde6-ccb4bc2b64da" path="/var/lib/kubelet/pods/cbb7a506-9811-4530-bde6-ccb4bc2b64da/volumes" Nov 03 00:10:37.584683 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:10:37.584644 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:10:40.489048 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:40.489002 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm"] Nov 03 00:10:40.489536 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:40.489268 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" podUID="fd3aa3e9-705b-4a34-a4af-b964c6991e7c" containerName="istio-proxy" containerID="cri-o://c7a3013c6eb6abf99fd3c7d8fbb88d6945dbef1dbbfc6fa607b97a9c13bf990f" gracePeriod=30 Nov 03 00:10:45.637914 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.637877 5114 generic.go:356] "Generic (PLEG): container finished" podID="fd3aa3e9-705b-4a34-a4af-b964c6991e7c" containerID="c7a3013c6eb6abf99fd3c7d8fbb88d6945dbef1dbbfc6fa607b97a9c13bf990f" exitCode=0 Nov 03 00:10:45.638358 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.637951 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" event={"ID":"fd3aa3e9-705b-4a34-a4af-b964c6991e7c","Type":"ContainerDied","Data":"c7a3013c6eb6abf99fd3c7d8fbb88d6945dbef1dbbfc6fa607b97a9c13bf990f"} Nov 03 00:10:45.745546 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.745510 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:45.836497 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.836459 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-token\" (UniqueName: \"kubernetes.io/projected/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-token\") pod \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " Nov 03 00:10:45.836729 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.836523 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-podinfo\") pod \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " Nov 03 00:10:45.836729 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.836549 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istiod-ca-cert\") pod \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " Nov 03 00:10:45.836729 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.836627 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-workload-certs\") pod \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " Nov 03 00:10:45.836729 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.836661 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-credential-socket\") pod \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " Nov 03 00:10:45.836729 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.836676 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-envoy\") pod \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " Nov 03 00:10:45.837053 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.836730 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-kube-api-access-76sc7\" (UniqueName: \"kubernetes.io/projected/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-kube-api-access-76sc7\") pod \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " Nov 03 00:10:45.837053 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.836765 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-workload-socket\") pod \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " Nov 03 00:10:45.837053 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.836820 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-data\") pod \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\" (UID: \"fd3aa3e9-705b-4a34-a4af-b964c6991e7c\") " Nov 03 00:10:45.837053 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.837027 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-workload-certs" pod "fd3aa3e9-705b-4a34-a4af-b964c6991e7c" (UID: "fd3aa3e9-705b-4a34-a4af-b964c6991e7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:10:45.837207 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.837155 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-credential-socket" pod "fd3aa3e9-705b-4a34-a4af-b964c6991e7c" (UID: "fd3aa3e9-705b-4a34-a4af-b964c6991e7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:10:45.837261 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.837235 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:45.837307 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.837281 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-data" pod "fd3aa3e9-705b-4a34-a4af-b964c6991e7c" (UID: "fd3aa3e9-705b-4a34-a4af-b964c6991e7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:10:45.837540 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.837412 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istiod-ca-cert" pod "fd3aa3e9-705b-4a34-a4af-b964c6991e7c" (UID: "fd3aa3e9-705b-4a34-a4af-b964c6991e7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:10:45.837540 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.837524 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-workload-socket" pod "fd3aa3e9-705b-4a34-a4af-b964c6991e7c" (UID: "fd3aa3e9-705b-4a34-a4af-b964c6991e7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:10:45.839136 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.839110 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-podinfo" pod "fd3aa3e9-705b-4a34-a4af-b964c6991e7c" (UID: "fd3aa3e9-705b-4a34-a4af-b964c6991e7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:10:45.839413 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.839392 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-kube-api-access-76sc7" pod "fd3aa3e9-705b-4a34-a4af-b964c6991e7c" (UID: "fd3aa3e9-705b-4a34-a4af-b964c6991e7c"). InnerVolumeSpecName "kube-api-access-76sc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:10:45.839602 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.839585 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-token" pod "fd3aa3e9-705b-4a34-a4af-b964c6991e7c" (UID: "fd3aa3e9-705b-4a34-a4af-b964c6991e7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:10:45.839729 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.839714 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-envoy" pod "fd3aa3e9-705b-4a34-a4af-b964c6991e7c" (UID: "fd3aa3e9-705b-4a34-a4af-b964c6991e7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:10:45.938498 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.938432 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:45.938697 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.938505 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:45.938697 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.938524 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76sc7\" (UniqueName: \"kubernetes.io/projected/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-kube-api-access-76sc7\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:45.938697 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.938540 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:45.938697 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.938553 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:45.938697 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.938567 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:45.938697 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.938581 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:45.938697 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:45.938595 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3aa3e9-705b-4a34-a4af-b964c6991e7c-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:46.046964 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:46.046923 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0a1c13d-xcv5j"] Nov 03 00:10:46.049539 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:46.049498 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0a1c13d-xcv5j"] Nov 03 00:10:46.644397 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:46.644364 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" Nov 03 00:10:46.644909 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:46.644391 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm" event={"ID":"fd3aa3e9-705b-4a34-a4af-b964c6991e7c","Type":"ContainerDied","Data":"2af48f25bb09848fc78e6a82c7a9f5d89124d73a3bba4caca2a85134188dcefa"} Nov 03 00:10:46.644909 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:46.644456 5114 scope.go:122] "RemoveContainer" containerID="c7a3013c6eb6abf99fd3c7d8fbb88d6945dbef1dbbfc6fa607b97a9c13bf990f" Nov 03 00:10:46.793416 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:46.793378 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65ec560e-262c-4f97-836e-ccdaeb231102" path="/var/lib/kubelet/pods/65ec560e-262c-4f97-836e-ccdaeb231102/volumes" Nov 03 00:10:46.858084 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:46.858042 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm"] Nov 03 00:10:46.907815 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:46.907720 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-hgcqm"] Nov 03 00:10:47.047077 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:47.047029 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7ab7aff-x8lgm"] Nov 03 00:10:47.050088 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:47.050058 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7ab7aff-x8lgm"] Nov 03 00:10:48.791998 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:48.791964 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6" path="/var/lib/kubelet/pods/741b89d4-3fa0-49e9-be5f-a5dcad5cc4c6/volumes" Nov 03 00:10:48.792423 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:48.792389 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd3aa3e9-705b-4a34-a4af-b964c6991e7c" path="/var/lib/kubelet/pods/fd3aa3e9-705b-4a34-a4af-b964c6991e7c/volumes" Nov 03 00:10:55.610244 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:55.607321 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7699-29939b"] Nov 03 00:10:55.610244 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:55.608295 5114 cpu_manager.go:414] "containerMap: removing container" podUID="fd3aa3e9-705b-4a34-a4af-b964c6991e7c" containerName="istio-proxy" Nov 03 00:10:55.610244 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:55.608318 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd3aa3e9-705b-4a34-a4af-b964c6991e7c" containerName="istio-proxy" Nov 03 00:10:55.610244 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:55.608541 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd3aa3e9-705b-4a34-a4af-b964c6991e7c" containerName="istio-proxy" Nov 03 00:10:55.614024 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:55.613996 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-29939b" Nov 03 00:10:55.665768 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:55.665729 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-29939b"] Nov 03 00:10:55.713367 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:55.713329 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-b8v4g\"" Nov 03 00:10:55.730206 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:55.730165 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2ph8\" (UniqueName: \"kubernetes.io/projected/7dee8758-1a4e-4b2c-8f83-de47352c5f07-kube-api-access-g2ph8\") pod \"test-curl-7699-29939b\" (UID: \"7dee8758-1a4e-4b2c-8f83-de47352c5f07\") " pod="opendatahub/test-curl-7699-29939b" Nov 03 00:10:55.831758 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:55.831718 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2ph8\" (UniqueName: \"kubernetes.io/projected/7dee8758-1a4e-4b2c-8f83-de47352c5f07-kube-api-access-g2ph8\") pod \"test-curl-7699-29939b\" (UID: \"7dee8758-1a4e-4b2c-8f83-de47352c5f07\") " pod="opendatahub/test-curl-7699-29939b" Nov 03 00:10:55.892282 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:55.892189 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2ph8\" (UniqueName: \"kubernetes.io/projected/7dee8758-1a4e-4b2c-8f83-de47352c5f07-kube-api-access-g2ph8\") pod \"test-curl-7699-29939b\" (UID: \"7dee8758-1a4e-4b2c-8f83-de47352c5f07\") " pod="opendatahub/test-curl-7699-29939b" Nov 03 00:10:55.927098 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:55.927060 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-29939b" Nov 03 00:10:56.299702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:56.299670 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-29939b"] Nov 03 00:10:56.302796 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:10:56.302754 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7dee8758_1a4e_4b2c_8f83_de47352c5f07.slice/crio-8e97604c1f63860c51a35cdac22266cf024be23468a00bfff91e84949cfecfc0 WatchSource:0}: Error finding container 8e97604c1f63860c51a35cdac22266cf024be23468a00bfff91e84949cfecfc0: Status 404 returned error can't find the container with id 8e97604c1f63860c51a35cdac22266cf024be23468a00bfff91e84949cfecfc0 Nov 03 00:10:56.698466 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:56.698342 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-29939b" event={"ID":"7dee8758-1a4e-4b2c-8f83-de47352c5f07","Type":"ContainerStarted","Data":"c09868f09dc422c6d5b0ba9a9d2bc024ffdcc8996a1c77d8d1bd04a76fe81efc"} Nov 03 00:10:56.698466 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:56.698409 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-29939b" event={"ID":"7dee8758-1a4e-4b2c-8f83-de47352c5f07","Type":"ContainerStarted","Data":"8e97604c1f63860c51a35cdac22266cf024be23468a00bfff91e84949cfecfc0"} Nov 03 00:10:56.811102 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:56.811028 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7699-29939b" podStartSLOduration=1.811007569 podStartE2EDuration="1.811007569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:56.810121943 +0000 UTC m=+4932.163155332" watchObservedRunningTime="2026-11-03 00:10:56.811007569 +0000 UTC m=+4932.164040940" Nov 03 00:10:58.291204 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:58.291156 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7699-29939b"] Nov 03 00:10:58.295578 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:58.295546 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7699-29939b"] Nov 03 00:10:58.446336 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:58.446298 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7699-5730ca"] Nov 03 00:10:58.446708 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:58.446693 5114 cpu_manager.go:414] "containerMap: removing container" podUID="7dee8758-1a4e-4b2c-8f83-de47352c5f07" containerName="test-curl-7699-29939b" Nov 03 00:10:58.446762 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:58.446710 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dee8758-1a4e-4b2c-8f83-de47352c5f07" containerName="test-curl-7699-29939b" Nov 03 00:10:58.446800 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:58.446773 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dee8758-1a4e-4b2c-8f83-de47352c5f07" containerName="test-curl-7699-29939b" Nov 03 00:10:58.449704 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:58.449682 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-5730ca" Nov 03 00:10:58.503371 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:58.503328 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-5730ca"] Nov 03 00:10:58.558686 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:58.558541 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn4fn\" (UniqueName: \"kubernetes.io/projected/ba9aec94-0fbb-4a5b-8bcd-d599f8a44672-kube-api-access-hn4fn\") pod \"test-curl-7699-5730ca\" (UID: \"ba9aec94-0fbb-4a5b-8bcd-d599f8a44672\") " pod="opendatahub/test-curl-7699-5730ca" Nov 03 00:10:58.660220 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:58.660176 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn4fn\" (UniqueName: \"kubernetes.io/projected/ba9aec94-0fbb-4a5b-8bcd-d599f8a44672-kube-api-access-hn4fn\") pod \"test-curl-7699-5730ca\" (UID: \"ba9aec94-0fbb-4a5b-8bcd-d599f8a44672\") " pod="opendatahub/test-curl-7699-5730ca" Nov 03 00:10:58.710431 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:58.710376 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7699-29939b" podUID="7dee8758-1a4e-4b2c-8f83-de47352c5f07" containerName="test-curl-7699-29939b" containerID="cri-o://c09868f09dc422c6d5b0ba9a9d2bc024ffdcc8996a1c77d8d1bd04a76fe81efc" gracePeriod=2 Nov 03 00:10:58.720844 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:58.720813 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn4fn\" (UniqueName: \"kubernetes.io/projected/ba9aec94-0fbb-4a5b-8bcd-d599f8a44672-kube-api-access-hn4fn\") pod \"test-curl-7699-5730ca\" (UID: \"ba9aec94-0fbb-4a5b-8bcd-d599f8a44672\") " pod="opendatahub/test-curl-7699-5730ca" Nov 03 00:10:58.761329 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:10:58.761288 5114 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7699-29939b\" is forbidden: User \"system:node:ip-10-0-135-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-250.us-west-1.compute.internal' and this object" podUID="7dee8758-1a4e-4b2c-8f83-de47352c5f07" pod="opendatahub/test-curl-7699-29939b" Nov 03 00:10:58.762319 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:58.762302 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-5730ca" Nov 03 00:10:59.134665 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:59.134637 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-5730ca"] Nov 03 00:10:59.137305 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:10:59.137272 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba9aec94_0fbb_4a5b_8bcd_d599f8a44672.slice/crio-5084637cf70ea009673949b3d1b686ced13ad2ef7076869160b501bab299cc2a WatchSource:0}: Error finding container 5084637cf70ea009673949b3d1b686ced13ad2ef7076869160b501bab299cc2a: Status 404 returned error can't find the container with id 5084637cf70ea009673949b3d1b686ced13ad2ef7076869160b501bab299cc2a Nov 03 00:10:59.717060 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:59.717017 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-5730ca" event={"ID":"ba9aec94-0fbb-4a5b-8bcd-d599f8a44672","Type":"ContainerStarted","Data":"9fde9d85b2c646323bdb80b6577b2dcc9becb1bb14e9c8ff898e747dd95dde17"} Nov 03 00:10:59.717513 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:10:59.717074 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-5730ca" event={"ID":"ba9aec94-0fbb-4a5b-8bcd-d599f8a44672","Type":"ContainerStarted","Data":"5084637cf70ea009673949b3d1b686ced13ad2ef7076869160b501bab299cc2a"} Nov 03 00:11:00.924756 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:00.924728 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-29939b" Nov 03 00:11:00.983548 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:00.983435 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dee8758-1a4e-4b2c-8f83-de47352c5f07-kube-api-access-g2ph8\" (UniqueName: \"kubernetes.io/projected/7dee8758-1a4e-4b2c-8f83-de47352c5f07-kube-api-access-g2ph8\") pod \"7dee8758-1a4e-4b2c-8f83-de47352c5f07\" (UID: \"7dee8758-1a4e-4b2c-8f83-de47352c5f07\") " Nov 03 00:11:00.986020 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:00.985977 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dee8758-1a4e-4b2c-8f83-de47352c5f07-kube-api-access-g2ph8" pod "7dee8758-1a4e-4b2c-8f83-de47352c5f07" (UID: "7dee8758-1a4e-4b2c-8f83-de47352c5f07"). InnerVolumeSpecName "kube-api-access-g2ph8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:11:01.084155 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:01.084120 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2ph8\" (UniqueName: \"kubernetes.io/projected/7dee8758-1a4e-4b2c-8f83-de47352c5f07-kube-api-access-g2ph8\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:01.253055 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:01.252936 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7699-5730ca" podStartSLOduration=3.2529183919999998 podStartE2EDuration="3.252918392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:59.829253077 +0000 UTC m=+4935.182286454" watchObservedRunningTime="2026-11-03 00:11:01.252918392 +0000 UTC m=+4936.605951767" Nov 03 00:11:01.254078 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:01.254056 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7699-5730ca"] Nov 03 00:11:01.258089 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:01.258056 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7699-5730ca"] Nov 03 00:11:01.728706 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:01.728668 5114 generic.go:356] "Generic (PLEG): container finished" podID="7dee8758-1a4e-4b2c-8f83-de47352c5f07" containerID="c09868f09dc422c6d5b0ba9a9d2bc024ffdcc8996a1c77d8d1bd04a76fe81efc" exitCode=137 Nov 03 00:11:01.728914 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:01.728721 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-29939b" Nov 03 00:11:01.728914 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:01.728769 5114 scope.go:122] "RemoveContainer" containerID="c09868f09dc422c6d5b0ba9a9d2bc024ffdcc8996a1c77d8d1bd04a76fe81efc" Nov 03 00:11:01.729066 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:01.729037 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7699-5730ca" podUID="ba9aec94-0fbb-4a5b-8bcd-d599f8a44672" containerName="test-curl-7699-5730ca" containerID="cri-o://9fde9d85b2c646323bdb80b6577b2dcc9becb1bb14e9c8ff898e747dd95dde17" gracePeriod=2 Nov 03 00:11:01.738199 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:01.738181 5114 scope.go:122] "RemoveContainer" containerID="c09868f09dc422c6d5b0ba9a9d2bc024ffdcc8996a1c77d8d1bd04a76fe81efc" Nov 03 00:11:01.738544 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:11:01.738519 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c09868f09dc422c6d5b0ba9a9d2bc024ffdcc8996a1c77d8d1bd04a76fe81efc\": container with ID starting with c09868f09dc422c6d5b0ba9a9d2bc024ffdcc8996a1c77d8d1bd04a76fe81efc not found: ID does not exist" containerID="c09868f09dc422c6d5b0ba9a9d2bc024ffdcc8996a1c77d8d1bd04a76fe81efc" Nov 03 00:11:01.738639 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:01.738553 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c09868f09dc422c6d5b0ba9a9d2bc024ffdcc8996a1c77d8d1bd04a76fe81efc"} err="failed to get container status \"c09868f09dc422c6d5b0ba9a9d2bc024ffdcc8996a1c77d8d1bd04a76fe81efc\": rpc error: code = NotFound desc = could not find container \"c09868f09dc422c6d5b0ba9a9d2bc024ffdcc8996a1c77d8d1bd04a76fe81efc\": container with ID starting with c09868f09dc422c6d5b0ba9a9d2bc024ffdcc8996a1c77d8d1bd04a76fe81efc not found: ID does not exist" Nov 03 00:11:01.779941 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:11:01.779906 5114 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7699-5730ca\" is forbidden: User \"system:node:ip-10-0-135-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-250.us-west-1.compute.internal' and this object" podUID="ba9aec94-0fbb-4a5b-8bcd-d599f8a44672" pod="opendatahub/test-curl-7699-5730ca" Nov 03 00:11:01.830822 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:11:01.830777 5114 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7699-29939b\" is forbidden: User \"system:node:ip-10-0-135-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-250.us-west-1.compute.internal' and this object" podUID="7dee8758-1a4e-4b2c-8f83-de47352c5f07" pod="opendatahub/test-curl-7699-29939b" Nov 03 00:11:01.881760 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:11:01.881711 5114 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7699-5730ca\" is forbidden: User \"system:node:ip-10-0-135-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-250.us-west-1.compute.internal' and this object" podUID="ba9aec94-0fbb-4a5b-8bcd-d599f8a44672" pod="opendatahub/test-curl-7699-5730ca" Nov 03 00:11:02.288094 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:02.288057 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad173b80-d5866"] Nov 03 00:11:02.288554 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:02.288434 5114 cpu_manager.go:414] "containerMap: removing container" podUID="ba9aec94-0fbb-4a5b-8bcd-d599f8a44672" containerName="test-curl-7699-5730ca" Nov 03 00:11:02.288554 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:02.288446 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba9aec94-0fbb-4a5b-8bcd-d599f8a44672" containerName="test-curl-7699-5730ca" Nov 03 00:11:02.288554 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:02.288532 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba9aec94-0fbb-4a5b-8bcd-d599f8a44672" containerName="test-curl-7699-5730ca" Nov 03 00:11:02.292832 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:02.292813 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad173b80-d5866" Nov 03 00:11:02.294426 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:02.294399 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls4sd\" (UniqueName: \"kubernetes.io/projected/c779dfd9-c39c-47fc-b573-b87ac84fc3b0-kube-api-access-ls4sd\") pod \"maas-api-revoke-sub-ad173b80-d5866\" (UID: \"c779dfd9-c39c-47fc-b573-b87ac84fc3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad173b80-d5866" Nov 03 00:11:02.294553 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:02.294457 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c779dfd9-c39c-47fc-b573-b87ac84fc3b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad173b80-d5866\" (UID: \"c779dfd9-c39c-47fc-b573-b87ac84fc3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad173b80-d5866" Nov 03 00:11:02.294597 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:02.294560 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c779dfd9-c39c-47fc-b573-b87ac84fc3b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad173b80-d5866\" (UID: \"c779dfd9-c39c-47fc-b573-b87ac84fc3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad173b80-d5866" Nov 03 00:11:02.343885 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:11:02.343845 5114 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7699-5730ca\" is forbidden: User \"system:node:ip-10-0-135-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-250.us-west-1.compute.internal' and this object" podUID="ba9aec94-0fbb-4a5b-8bcd-d599f8a44672" pod="opendatahub/test-curl-7699-5730ca" Nov 03 00:11:02.347087 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:02.347014 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:11:02.349454 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:02.349413 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad173b80-d5866"] Nov 03 00:11:02.393561 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:02.393521 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 03 00:11:02.395129 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:02.395097 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls4sd\" (UniqueName: \"kubernetes.io/projected/c779dfd9-c39c-47fc-b573-b87ac84fc3b0-kube-api-access-ls4sd\") pod \"maas-api-revoke-sub-ad173b80-d5866\" (UID: \"c779dfd9-c39c-47fc-b573-b87ac84fc3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad173b80-d5866" Nov 03 00:11:02.395200 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:02.395169 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c779dfd9-c39c-47fc-b573-b87ac84fc3b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad173b80-d5866\" (UID: \"c779dfd9-c39c-47fc-b573-b87ac84fc3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad173b80-d5866" Nov 03 00:11:02.395200 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:02.395191 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c779dfd9-c39c-47fc-b573-b87ac84fc3b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad173b80-d5866\" (UID: \"c779dfd9-c39c-47fc-b573-b87ac84fc3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad173b80-d5866" Nov 03 00:11:02.395886 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:02.395857 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c779dfd9-c39c-47fc-b573-b87ac84fc3b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad173b80-d5866\" (UID: \"c779dfd9-c39c-47fc-b573-b87ac84fc3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad173b80-d5866" Nov 03 00:11:02.397935 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:02.397912 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c779dfd9-c39c-47fc-b573-b87ac84fc3b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad173b80-d5866\" (UID: \"c779dfd9-c39c-47fc-b573-b87ac84fc3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad173b80-d5866" Nov 03 00:11:02.455990 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:02.455949 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls4sd\" (UniqueName: \"kubernetes.io/projected/c779dfd9-c39c-47fc-b573-b87ac84fc3b0-kube-api-access-ls4sd\") pod \"maas-api-revoke-sub-ad173b80-d5866\" (UID: \"c779dfd9-c39c-47fc-b573-b87ac84fc3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad173b80-d5866" Nov 03 00:11:02.507719 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:11:02.507672 5114 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7699-29939b\" is forbidden: User \"system:node:ip-10-0-135-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-250.us-west-1.compute.internal' and this object" podUID="7dee8758-1a4e-4b2c-8f83-de47352c5f07" pod="opendatahub/test-curl-7699-29939b" Nov 03 00:11:02.606522 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:02.606389 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad173b80-d5866" Nov 03 00:11:02.792372 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:02.792336 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dee8758-1a4e-4b2c-8f83-de47352c5f07" path="/var/lib/kubelet/pods/7dee8758-1a4e-4b2c-8f83-de47352c5f07/volumes" Nov 03 00:11:02.981848 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:02.981576 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad173b80-d5866"] Nov 03 00:11:02.984259 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:11:02.984228 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc779dfd9_c39c_47fc_b573_b87ac84fc3b0.slice/crio-c2ab067c9a56cc78fc4350dbf59077f6b7e0efc1485365c937d108031f1b5475 WatchSource:0}: Error finding container c2ab067c9a56cc78fc4350dbf59077f6b7e0efc1485365c937d108031f1b5475: Status 404 returned error can't find the container with id c2ab067c9a56cc78fc4350dbf59077f6b7e0efc1485365c937d108031f1b5475 Nov 03 00:11:03.743774 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:03.743737 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad173b80-d5866" event={"ID":"c779dfd9-c39c-47fc-b573-b87ac84fc3b0","Type":"ContainerStarted","Data":"a062f6d35a2dce88549a3f864e340066a065144e311e356515e9bda929d06ab2"} Nov 03 00:11:03.744260 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:03.743795 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad173b80-d5866" event={"ID":"c779dfd9-c39c-47fc-b573-b87ac84fc3b0","Type":"ContainerStarted","Data":"c2ab067c9a56cc78fc4350dbf59077f6b7e0efc1485365c937d108031f1b5475"} Nov 03 00:11:03.859261 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:03.859201 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad173b80-d5866" podStartSLOduration=1.859182898 podStartE2EDuration="1.859182898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:11:03.857621613 +0000 UTC m=+4939.210655194" watchObservedRunningTime="2026-11-03 00:11:03.859182898 +0000 UTC m=+4939.212216274" Nov 03 00:11:03.946523 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:03.946461 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-5730ca" Nov 03 00:11:04.007821 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:04.007733 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba9aec94-0fbb-4a5b-8bcd-d599f8a44672-kube-api-access-hn4fn\" (UniqueName: \"kubernetes.io/projected/ba9aec94-0fbb-4a5b-8bcd-d599f8a44672-kube-api-access-hn4fn\") pod \"ba9aec94-0fbb-4a5b-8bcd-d599f8a44672\" (UID: \"ba9aec94-0fbb-4a5b-8bcd-d599f8a44672\") " Nov 03 00:11:04.010087 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:04.010048 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba9aec94-0fbb-4a5b-8bcd-d599f8a44672-kube-api-access-hn4fn" pod "ba9aec94-0fbb-4a5b-8bcd-d599f8a44672" (UID: "ba9aec94-0fbb-4a5b-8bcd-d599f8a44672"). InnerVolumeSpecName "kube-api-access-hn4fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:11:04.108769 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:04.108736 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn4fn\" (UniqueName: \"kubernetes.io/projected/ba9aec94-0fbb-4a5b-8bcd-d599f8a44672-kube-api-access-hn4fn\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:04.750500 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:04.750443 5114 generic.go:356] "Generic (PLEG): container finished" podID="c779dfd9-c39c-47fc-b573-b87ac84fc3b0" containerID="a062f6d35a2dce88549a3f864e340066a065144e311e356515e9bda929d06ab2" exitCode=0 Nov 03 00:11:04.750969 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:04.750534 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad173b80-d5866" event={"ID":"c779dfd9-c39c-47fc-b573-b87ac84fc3b0","Type":"ContainerDied","Data":"a062f6d35a2dce88549a3f864e340066a065144e311e356515e9bda929d06ab2"} Nov 03 00:11:04.751942 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:04.751919 5114 generic.go:356] "Generic (PLEG): container finished" podID="ba9aec94-0fbb-4a5b-8bcd-d599f8a44672" containerID="9fde9d85b2c646323bdb80b6577b2dcc9becb1bb14e9c8ff898e747dd95dde17" exitCode=137 Nov 03 00:11:04.752039 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:04.751971 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-5730ca" Nov 03 00:11:04.752039 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:04.752016 5114 scope.go:122] "RemoveContainer" containerID="9fde9d85b2c646323bdb80b6577b2dcc9becb1bb14e9c8ff898e747dd95dde17" Nov 03 00:11:04.761734 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:04.761713 5114 scope.go:122] "RemoveContainer" containerID="9fde9d85b2c646323bdb80b6577b2dcc9becb1bb14e9c8ff898e747dd95dde17" Nov 03 00:11:04.762046 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:11:04.762022 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fde9d85b2c646323bdb80b6577b2dcc9becb1bb14e9c8ff898e747dd95dde17\": container with ID starting with 9fde9d85b2c646323bdb80b6577b2dcc9becb1bb14e9c8ff898e747dd95dde17 not found: ID does not exist" containerID="9fde9d85b2c646323bdb80b6577b2dcc9becb1bb14e9c8ff898e747dd95dde17" Nov 03 00:11:04.762099 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:04.762057 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fde9d85b2c646323bdb80b6577b2dcc9becb1bb14e9c8ff898e747dd95dde17"} err="failed to get container status \"9fde9d85b2c646323bdb80b6577b2dcc9becb1bb14e9c8ff898e747dd95dde17\": rpc error: code = NotFound desc = could not find container \"9fde9d85b2c646323bdb80b6577b2dcc9becb1bb14e9c8ff898e747dd95dde17\": container with ID starting with 9fde9d85b2c646323bdb80b6577b2dcc9becb1bb14e9c8ff898e747dd95dde17 not found: ID does not exist" Nov 03 00:11:04.792465 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:04.792427 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba9aec94-0fbb-4a5b-8bcd-d599f8a44672" path="/var/lib/kubelet/pods/ba9aec94-0fbb-4a5b-8bcd-d599f8a44672/volumes" Nov 03 00:11:05.888551 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:05.888523 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad173b80-d5866" Nov 03 00:11:05.927110 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:05.927070 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c779dfd9-c39c-47fc-b573-b87ac84fc3b0-kube-api-access-ls4sd\" (UniqueName: \"kubernetes.io/projected/c779dfd9-c39c-47fc-b573-b87ac84fc3b0-kube-api-access-ls4sd\") pod \"c779dfd9-c39c-47fc-b573-b87ac84fc3b0\" (UID: \"c779dfd9-c39c-47fc-b573-b87ac84fc3b0\") " Nov 03 00:11:05.927331 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:05.927130 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c779dfd9-c39c-47fc-b573-b87ac84fc3b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c779dfd9-c39c-47fc-b573-b87ac84fc3b0-maas-api-xks-ca\") pod \"c779dfd9-c39c-47fc-b573-b87ac84fc3b0\" (UID: \"c779dfd9-c39c-47fc-b573-b87ac84fc3b0\") " Nov 03 00:11:05.927331 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:05.927179 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c779dfd9-c39c-47fc-b573-b87ac84fc3b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c779dfd9-c39c-47fc-b573-b87ac84fc3b0-maas-api-openshift-ca\") pod \"c779dfd9-c39c-47fc-b573-b87ac84fc3b0\" (UID: \"c779dfd9-c39c-47fc-b573-b87ac84fc3b0\") " Nov 03 00:11:05.927572 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:05.927546 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c779dfd9-c39c-47fc-b573-b87ac84fc3b0-maas-api-openshift-ca" pod "c779dfd9-c39c-47fc-b573-b87ac84fc3b0" (UID: "c779dfd9-c39c-47fc-b573-b87ac84fc3b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:11:05.929428 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:05.929404 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c779dfd9-c39c-47fc-b573-b87ac84fc3b0-maas-api-xks-ca" pod "c779dfd9-c39c-47fc-b573-b87ac84fc3b0" (UID: "c779dfd9-c39c-47fc-b573-b87ac84fc3b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:11:05.929549 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:05.929464 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c779dfd9-c39c-47fc-b573-b87ac84fc3b0-kube-api-access-ls4sd" pod "c779dfd9-c39c-47fc-b573-b87ac84fc3b0" (UID: "c779dfd9-c39c-47fc-b573-b87ac84fc3b0"). InnerVolumeSpecName "kube-api-access-ls4sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:11:06.027933 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:06.027823 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c779dfd9-c39c-47fc-b573-b87ac84fc3b0-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:06.027933 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:06.027860 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c779dfd9-c39c-47fc-b573-b87ac84fc3b0-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:06.027933 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:06.027872 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls4sd\" (UniqueName: \"kubernetes.io/projected/c779dfd9-c39c-47fc-b573-b87ac84fc3b0-kube-api-access-ls4sd\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:06.157061 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:06.157020 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7997331e-mhn2m"] Nov 03 00:11:06.157426 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:06.157412 5114 cpu_manager.go:414] "containerMap: removing container" podUID="c779dfd9-c39c-47fc-b573-b87ac84fc3b0" containerName="revoke-keys" Nov 03 00:11:06.157473 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:06.157427 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c779dfd9-c39c-47fc-b573-b87ac84fc3b0" containerName="revoke-keys" Nov 03 00:11:06.157533 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:06.157524 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="c779dfd9-c39c-47fc-b573-b87ac84fc3b0" containerName="revoke-keys" Nov 03 00:11:06.160704 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:06.160679 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7997331e-mhn2m" Nov 03 00:11:06.218653 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:06.218611 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7997331e-mhn2m"] Nov 03 00:11:06.229882 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:06.229841 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66555e84-a411-4dc1-ae33-d585c63e96d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7997331e-mhn2m\" (UID: \"66555e84-a411-4dc1-ae33-d585c63e96d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7997331e-mhn2m" Nov 03 00:11:06.230098 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:06.229898 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66555e84-a411-4dc1-ae33-d585c63e96d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7997331e-mhn2m\" (UID: \"66555e84-a411-4dc1-ae33-d585c63e96d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7997331e-mhn2m" Nov 03 00:11:06.230098 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:06.229960 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl8hg\" (UniqueName: \"kubernetes.io/projected/66555e84-a411-4dc1-ae33-d585c63e96d2-kube-api-access-xl8hg\") pod \"maas-api-revoke-sub-7997331e-mhn2m\" (UID: \"66555e84-a411-4dc1-ae33-d585c63e96d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7997331e-mhn2m" Nov 03 00:11:06.330868 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:06.330827 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66555e84-a411-4dc1-ae33-d585c63e96d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7997331e-mhn2m\" (UID: \"66555e84-a411-4dc1-ae33-d585c63e96d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7997331e-mhn2m" Nov 03 00:11:06.331090 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:06.330892 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66555e84-a411-4dc1-ae33-d585c63e96d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7997331e-mhn2m\" (UID: \"66555e84-a411-4dc1-ae33-d585c63e96d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7997331e-mhn2m" Nov 03 00:11:06.331090 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:06.330913 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl8hg\" (UniqueName: \"kubernetes.io/projected/66555e84-a411-4dc1-ae33-d585c63e96d2-kube-api-access-xl8hg\") pod \"maas-api-revoke-sub-7997331e-mhn2m\" (UID: \"66555e84-a411-4dc1-ae33-d585c63e96d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7997331e-mhn2m" Nov 03 00:11:06.331671 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:06.331647 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66555e84-a411-4dc1-ae33-d585c63e96d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7997331e-mhn2m\" (UID: \"66555e84-a411-4dc1-ae33-d585c63e96d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7997331e-mhn2m" Nov 03 00:11:06.333669 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:06.333643 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66555e84-a411-4dc1-ae33-d585c63e96d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7997331e-mhn2m\" (UID: \"66555e84-a411-4dc1-ae33-d585c63e96d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7997331e-mhn2m" Nov 03 00:11:06.391602 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:06.391564 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl8hg\" (UniqueName: \"kubernetes.io/projected/66555e84-a411-4dc1-ae33-d585c63e96d2-kube-api-access-xl8hg\") pod \"maas-api-revoke-sub-7997331e-mhn2m\" (UID: \"66555e84-a411-4dc1-ae33-d585c63e96d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7997331e-mhn2m" Nov 03 00:11:06.474649 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:06.474613 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7997331e-mhn2m" Nov 03 00:11:06.764674 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:06.764591 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad173b80-d5866" Nov 03 00:11:06.764841 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:06.764584 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad173b80-d5866" event={"ID":"c779dfd9-c39c-47fc-b573-b87ac84fc3b0","Type":"ContainerDied","Data":"c2ab067c9a56cc78fc4350dbf59077f6b7e0efc1485365c937d108031f1b5475"} Nov 03 00:11:06.764841 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:06.764716 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2ab067c9a56cc78fc4350dbf59077f6b7e0efc1485365c937d108031f1b5475" Nov 03 00:11:06.845567 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:06.845537 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7997331e-mhn2m"] Nov 03 00:11:06.848007 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:11:06.847975 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66555e84_a411_4dc1_ae33_d585c63e96d2.slice/crio-37bd02e9dee5f3f40e7341e5a9406500019b8f4059e5445fe8cd1fc5fddc26ff WatchSource:0}: Error finding container 37bd02e9dee5f3f40e7341e5a9406500019b8f4059e5445fe8cd1fc5fddc26ff: Status 404 returned error can't find the container with id 37bd02e9dee5f3f40e7341e5a9406500019b8f4059e5445fe8cd1fc5fddc26ff Nov 03 00:11:07.583749 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:11:07.583710 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:11:07.771445 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:07.771407 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7997331e-mhn2m" event={"ID":"66555e84-a411-4dc1-ae33-d585c63e96d2","Type":"ContainerStarted","Data":"2abf2314d589c747f5abaa882754f43d2ced4cc2a440b89291c7c484cb5b9358"} Nov 03 00:11:07.771719 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:07.771462 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7997331e-mhn2m" event={"ID":"66555e84-a411-4dc1-ae33-d585c63e96d2","Type":"ContainerStarted","Data":"37bd02e9dee5f3f40e7341e5a9406500019b8f4059e5445fe8cd1fc5fddc26ff"} Nov 03 00:11:07.886504 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:07.886366 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7997331e-mhn2m" podStartSLOduration=1.8863455519999999 podStartE2EDuration="1.886345552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:11:07.88586774 +0000 UTC m=+4943.238901117" watchObservedRunningTime="2026-11-03 00:11:07.886345552 +0000 UTC m=+4943.239378928" Nov 03 00:11:08.778248 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:08.778208 5114 generic.go:356] "Generic (PLEG): container finished" podID="66555e84-a411-4dc1-ae33-d585c63e96d2" containerID="2abf2314d589c747f5abaa882754f43d2ced4cc2a440b89291c7c484cb5b9358" exitCode=0 Nov 03 00:11:08.778746 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:08.778293 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7997331e-mhn2m" event={"ID":"66555e84-a411-4dc1-ae33-d585c63e96d2","Type":"ContainerDied","Data":"2abf2314d589c747f5abaa882754f43d2ced4cc2a440b89291c7c484cb5b9358"} Nov 03 00:11:09.932308 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:09.932278 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7997331e-mhn2m" Nov 03 00:11:09.967002 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:09.966963 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66555e84-a411-4dc1-ae33-d585c63e96d2-kube-api-access-xl8hg\" (UniqueName: \"kubernetes.io/projected/66555e84-a411-4dc1-ae33-d585c63e96d2-kube-api-access-xl8hg\") pod \"66555e84-a411-4dc1-ae33-d585c63e96d2\" (UID: \"66555e84-a411-4dc1-ae33-d585c63e96d2\") " Nov 03 00:11:09.967225 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:09.967029 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66555e84-a411-4dc1-ae33-d585c63e96d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66555e84-a411-4dc1-ae33-d585c63e96d2-maas-api-xks-ca\") pod \"66555e84-a411-4dc1-ae33-d585c63e96d2\" (UID: \"66555e84-a411-4dc1-ae33-d585c63e96d2\") " Nov 03 00:11:09.967225 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:09.967115 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66555e84-a411-4dc1-ae33-d585c63e96d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66555e84-a411-4dc1-ae33-d585c63e96d2-maas-api-openshift-ca\") pod \"66555e84-a411-4dc1-ae33-d585c63e96d2\" (UID: \"66555e84-a411-4dc1-ae33-d585c63e96d2\") " Nov 03 00:11:09.967527 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:09.967472 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66555e84-a411-4dc1-ae33-d585c63e96d2-maas-api-openshift-ca" pod "66555e84-a411-4dc1-ae33-d585c63e96d2" (UID: "66555e84-a411-4dc1-ae33-d585c63e96d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:11:09.969350 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:09.969327 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66555e84-a411-4dc1-ae33-d585c63e96d2-kube-api-access-xl8hg" pod "66555e84-a411-4dc1-ae33-d585c63e96d2" (UID: "66555e84-a411-4dc1-ae33-d585c63e96d2"). InnerVolumeSpecName "kube-api-access-xl8hg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:11:09.969431 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:09.969401 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66555e84-a411-4dc1-ae33-d585c63e96d2-maas-api-xks-ca" pod "66555e84-a411-4dc1-ae33-d585c63e96d2" (UID: "66555e84-a411-4dc1-ae33-d585c63e96d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:11:10.068514 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:10.068386 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66555e84-a411-4dc1-ae33-d585c63e96d2-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:10.068514 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:10.068422 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl8hg\" (UniqueName: \"kubernetes.io/projected/66555e84-a411-4dc1-ae33-d585c63e96d2-kube-api-access-xl8hg\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:10.068514 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:10.068438 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66555e84-a411-4dc1-ae33-d585c63e96d2-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:10.789516 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:10.789472 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7997331e-mhn2m" Nov 03 00:11:10.792127 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:10.792098 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7997331e-mhn2m" event={"ID":"66555e84-a411-4dc1-ae33-d585c63e96d2","Type":"ContainerDied","Data":"37bd02e9dee5f3f40e7341e5a9406500019b8f4059e5445fe8cd1fc5fddc26ff"} Nov 03 00:11:10.792285 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:10.792144 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37bd02e9dee5f3f40e7341e5a9406500019b8f4059e5445fe8cd1fc5fddc26ff" Nov 03 00:11:12.593379 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.593344 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v"] Nov 03 00:11:12.593831 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.593700 5114 cpu_manager.go:414] "containerMap: removing container" podUID="66555e84-a411-4dc1-ae33-d585c63e96d2" containerName="revoke-keys" Nov 03 00:11:12.593831 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.593712 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66555e84-a411-4dc1-ae33-d585c63e96d2" containerName="revoke-keys" Nov 03 00:11:12.593831 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.593789 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="66555e84-a411-4dc1-ae33-d585c63e96d2" containerName="revoke-keys" Nov 03 00:11:12.596942 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.596921 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.649285 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.649246 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-b8d3cb-openshift-default-dockercfg-4rl92\"" Nov 03 00:11:12.693898 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.693856 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-istio-data\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.694082 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.693902 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-istio-envoy\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.694082 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.693986 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-workload-socket\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.694082 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.694022 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpxbg\" (UniqueName: \"kubernetes.io/projected/14d58c32-6605-4191-a4cd-3a311461a25a-kube-api-access-xpxbg\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.694082 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.694067 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14d58c32-6605-4191-a4cd-3a311461a25a-istiod-ca-cert\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.694232 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.694168 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-credential-socket\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.694232 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.694222 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14d58c32-6605-4191-a4cd-3a311461a25a-istio-token\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.694299 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.694247 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-workload-certs\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.694337 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.694308 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14d58c32-6605-4191-a4cd-3a311461a25a-istio-podinfo\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.701073 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.701036 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v"] Nov 03 00:11:12.795167 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.795129 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14d58c32-6605-4191-a4cd-3a311461a25a-istio-token\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.795398 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.795177 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-workload-certs\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.795398 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.795209 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14d58c32-6605-4191-a4cd-3a311461a25a-istio-podinfo\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.795398 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.795251 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-istio-data\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.795398 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.795285 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-istio-envoy\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.795398 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.795341 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-workload-socket\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.795769 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.795717 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-workload-certs\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.795769 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.795752 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-istio-data\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.795890 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.795841 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpxbg\" (UniqueName: \"kubernetes.io/projected/14d58c32-6605-4191-a4cd-3a311461a25a-kube-api-access-xpxbg\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.795890 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.795867 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-workload-socket\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.795992 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.795936 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14d58c32-6605-4191-a4cd-3a311461a25a-istiod-ca-cert\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.796046 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.795989 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-credential-socket\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.796326 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.796282 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-credential-socket\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.796983 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.796932 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14d58c32-6605-4191-a4cd-3a311461a25a-istiod-ca-cert\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.798303 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.798273 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-istio-envoy\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.798419 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.798407 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14d58c32-6605-4191-a4cd-3a311461a25a-istio-podinfo\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.856380 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.856308 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpxbg\" (UniqueName: \"kubernetes.io/projected/14d58c32-6605-4191-a4cd-3a311461a25a-kube-api-access-xpxbg\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.856380 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.856308 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14d58c32-6605-4191-a4cd-3a311461a25a-istio-token\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:12.910063 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:12.910026 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:13.113156 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:13.113069 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5616f809-l5dnn"] Nov 03 00:11:13.117334 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:13.117314 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5616f809-l5dnn" Nov 03 00:11:13.172093 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:13.172034 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 03 00:11:13.172093 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:13.172045 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:11:13.174745 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:13.174711 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5616f809-l5dnn"] Nov 03 00:11:13.199954 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:13.199909 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhllz\" (UniqueName: \"kubernetes.io/projected/ee486b7f-543e-4475-840c-103833c83b2b-kube-api-access-jhllz\") pod \"maas-api-revoke-sub-5616f809-l5dnn\" (UID: \"ee486b7f-543e-4475-840c-103833c83b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5616f809-l5dnn" Nov 03 00:11:13.200127 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:13.199970 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee486b7f-543e-4475-840c-103833c83b2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5616f809-l5dnn\" (UID: \"ee486b7f-543e-4475-840c-103833c83b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5616f809-l5dnn" Nov 03 00:11:13.200127 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:13.200071 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee486b7f-543e-4475-840c-103833c83b2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5616f809-l5dnn\" (UID: \"ee486b7f-543e-4475-840c-103833c83b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5616f809-l5dnn" Nov 03 00:11:13.301146 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:13.301098 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee486b7f-543e-4475-840c-103833c83b2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5616f809-l5dnn\" (UID: \"ee486b7f-543e-4475-840c-103833c83b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5616f809-l5dnn" Nov 03 00:11:13.301347 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:13.301212 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhllz\" (UniqueName: \"kubernetes.io/projected/ee486b7f-543e-4475-840c-103833c83b2b-kube-api-access-jhllz\") pod \"maas-api-revoke-sub-5616f809-l5dnn\" (UID: \"ee486b7f-543e-4475-840c-103833c83b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5616f809-l5dnn" Nov 03 00:11:13.301347 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:13.301246 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee486b7f-543e-4475-840c-103833c83b2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5616f809-l5dnn\" (UID: \"ee486b7f-543e-4475-840c-103833c83b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5616f809-l5dnn" Nov 03 00:11:13.301869 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:13.301846 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee486b7f-543e-4475-840c-103833c83b2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5616f809-l5dnn\" (UID: \"ee486b7f-543e-4475-840c-103833c83b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5616f809-l5dnn" Nov 03 00:11:13.303840 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:13.303819 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee486b7f-543e-4475-840c-103833c83b2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5616f809-l5dnn\" (UID: \"ee486b7f-543e-4475-840c-103833c83b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5616f809-l5dnn" Nov 03 00:11:13.344527 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:13.344476 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v"] Nov 03 00:11:13.347008 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:11:13.346977 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d58c32_6605_4191_a4cd_3a311461a25a.slice/crio-2f8bfc6766c3d810a4cf2dfb94147421ea4b761406394c0f552d124b2b78c52e WatchSource:0}: Error finding container 2f8bfc6766c3d810a4cf2dfb94147421ea4b761406394c0f552d124b2b78c52e: Status 404 returned error can't find the container with id 2f8bfc6766c3d810a4cf2dfb94147421ea4b761406394c0f552d124b2b78c52e Nov 03 00:11:13.349417 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:13.349379 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:11:13.349577 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:13.349469 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:11:13.349577 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:13.349531 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:11:13.363647 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:13.363588 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhllz\" (UniqueName: \"kubernetes.io/projected/ee486b7f-543e-4475-840c-103833c83b2b-kube-api-access-jhllz\") pod \"maas-api-revoke-sub-5616f809-l5dnn\" (UID: \"ee486b7f-543e-4475-840c-103833c83b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5616f809-l5dnn" Nov 03 00:11:13.430396 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:13.430359 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5616f809-l5dnn" Nov 03 00:11:13.812651 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:13.812607 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5616f809-l5dnn"] Nov 03 00:11:13.817074 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:13.816336 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" event={"ID":"14d58c32-6605-4191-a4cd-3a311461a25a","Type":"ContainerStarted","Data":"b11e70f7184e27f11fb038748dd53f54b6e19253b2b5e34b976f8a06b8df3c0a"} Nov 03 00:11:13.817074 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:13.816402 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" event={"ID":"14d58c32-6605-4191-a4cd-3a311461a25a","Type":"ContainerStarted","Data":"2f8bfc6766c3d810a4cf2dfb94147421ea4b761406394c0f552d124b2b78c52e"} Nov 03 00:11:13.823572 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:11:13.819184 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee486b7f_543e_4475_840c_103833c83b2b.slice/crio-9ca3f7ad4158c582677032fdcaeb4c84c0dc8bb4f8c04c7d3e77e40a50955bd5 WatchSource:0}: Error finding container 9ca3f7ad4158c582677032fdcaeb4c84c0dc8bb4f8c04c7d3e77e40a50955bd5: Status 404 returned error can't find the container with id 9ca3f7ad4158c582677032fdcaeb4c84c0dc8bb4f8c04c7d3e77e40a50955bd5 Nov 03 00:11:13.920645 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:13.920590 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:13.936609 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:13.936526 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" podStartSLOduration=1.9365055230000001 podStartE2EDuration="1.936505523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:11:13.935222726 +0000 UTC m=+4949.288256103" watchObservedRunningTime="2026-11-03 00:11:13.936505523 +0000 UTC m=+4949.289538899" Nov 03 00:11:14.835181 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:14.835129 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5616f809-l5dnn" event={"ID":"ee486b7f-543e-4475-840c-103833c83b2b","Type":"ContainerStarted","Data":"9f758be5783e37369cacbc8075f6c06a9620d44396349ca30a78b509810b0dd0"} Nov 03 00:11:14.835735 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:14.835198 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5616f809-l5dnn" event={"ID":"ee486b7f-543e-4475-840c-103833c83b2b","Type":"ContainerStarted","Data":"9ca3f7ad4158c582677032fdcaeb4c84c0dc8bb4f8c04c7d3e77e40a50955bd5"} Nov 03 00:11:14.932360 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:14.932324 5114 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 03 00:11:14.932585 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:14.932380 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" podUID="14d58c32-6605-4191-a4cd-3a311461a25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 03 00:11:15.000418 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:15.000362 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5616f809-l5dnn" podStartSLOduration=2.000347676 podStartE2EDuration="2.000347676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:11:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:11:14.949797976 +0000 UTC m=+4950.302831353" watchObservedRunningTime="2026-11-03 00:11:15.000347676 +0000 UTC m=+4950.353381051" Nov 03 00:11:15.842889 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:15.842853 5114 generic.go:356] "Generic (PLEG): container finished" podID="ee486b7f-543e-4475-840c-103833c83b2b" containerID="9f758be5783e37369cacbc8075f6c06a9620d44396349ca30a78b509810b0dd0" exitCode=0 Nov 03 00:11:15.843332 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:15.842934 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5616f809-l5dnn" event={"ID":"ee486b7f-543e-4475-840c-103833c83b2b","Type":"ContainerDied","Data":"9f758be5783e37369cacbc8075f6c06a9620d44396349ca30a78b509810b0dd0"} Nov 03 00:11:15.926511 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:15.912930 5114 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 03 00:11:15.926511 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:15.913014 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" podUID="14d58c32-6605-4191-a4cd-3a311461a25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 03 00:11:16.844660 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:16.844626 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9948ecf8-wg2ls"] Nov 03 00:11:16.848819 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:16.848282 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9948ecf8-wg2ls" Nov 03 00:11:16.921048 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:16.905954 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9948ecf8-wg2ls"] Nov 03 00:11:16.935346 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:16.934732 5114 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 03 00:11:16.935346 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:16.934797 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" podUID="14d58c32-6605-4191-a4cd-3a311461a25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 03 00:11:16.960619 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:16.959758 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e916d42-6ee5-41dd-a6c6-c331286e11d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9948ecf8-wg2ls\" (UID: \"2e916d42-6ee5-41dd-a6c6-c331286e11d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9948ecf8-wg2ls" Nov 03 00:11:16.960619 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:16.959829 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpfh4\" (UniqueName: \"kubernetes.io/projected/2e916d42-6ee5-41dd-a6c6-c331286e11d3-kube-api-access-lpfh4\") pod \"maas-api-revoke-sub-9948ecf8-wg2ls\" (UID: \"2e916d42-6ee5-41dd-a6c6-c331286e11d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9948ecf8-wg2ls" Nov 03 00:11:16.960619 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:16.959946 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e916d42-6ee5-41dd-a6c6-c331286e11d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9948ecf8-wg2ls\" (UID: \"2e916d42-6ee5-41dd-a6c6-c331286e11d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9948ecf8-wg2ls" Nov 03 00:11:17.041896 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.041867 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5616f809-l5dnn" Nov 03 00:11:17.060581 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.060530 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e916d42-6ee5-41dd-a6c6-c331286e11d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9948ecf8-wg2ls\" (UID: \"2e916d42-6ee5-41dd-a6c6-c331286e11d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9948ecf8-wg2ls" Nov 03 00:11:17.060768 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.060651 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e916d42-6ee5-41dd-a6c6-c331286e11d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9948ecf8-wg2ls\" (UID: \"2e916d42-6ee5-41dd-a6c6-c331286e11d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9948ecf8-wg2ls" Nov 03 00:11:17.060768 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.060699 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpfh4\" (UniqueName: \"kubernetes.io/projected/2e916d42-6ee5-41dd-a6c6-c331286e11d3-kube-api-access-lpfh4\") pod \"maas-api-revoke-sub-9948ecf8-wg2ls\" (UID: \"2e916d42-6ee5-41dd-a6c6-c331286e11d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9948ecf8-wg2ls" Nov 03 00:11:17.061290 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.061267 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e916d42-6ee5-41dd-a6c6-c331286e11d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9948ecf8-wg2ls\" (UID: \"2e916d42-6ee5-41dd-a6c6-c331286e11d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9948ecf8-wg2ls" Nov 03 00:11:17.063412 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.063390 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e916d42-6ee5-41dd-a6c6-c331286e11d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9948ecf8-wg2ls\" (UID: \"2e916d42-6ee5-41dd-a6c6-c331286e11d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9948ecf8-wg2ls" Nov 03 00:11:17.131677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.131578 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpfh4\" (UniqueName: \"kubernetes.io/projected/2e916d42-6ee5-41dd-a6c6-c331286e11d3-kube-api-access-lpfh4\") pod \"maas-api-revoke-sub-9948ecf8-wg2ls\" (UID: \"2e916d42-6ee5-41dd-a6c6-c331286e11d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9948ecf8-wg2ls" Nov 03 00:11:17.163560 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.161398 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee486b7f-543e-4475-840c-103833c83b2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee486b7f-543e-4475-840c-103833c83b2b-maas-api-xks-ca\") pod \"ee486b7f-543e-4475-840c-103833c83b2b\" (UID: \"ee486b7f-543e-4475-840c-103833c83b2b\") " Nov 03 00:11:17.163560 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.161536 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9948ecf8-wg2ls" Nov 03 00:11:17.163560 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.161559 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee486b7f-543e-4475-840c-103833c83b2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee486b7f-543e-4475-840c-103833c83b2b-maas-api-openshift-ca\") pod \"ee486b7f-543e-4475-840c-103833c83b2b\" (UID: \"ee486b7f-543e-4475-840c-103833c83b2b\") " Nov 03 00:11:17.163560 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.161591 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee486b7f-543e-4475-840c-103833c83b2b-kube-api-access-jhllz\" (UniqueName: \"kubernetes.io/projected/ee486b7f-543e-4475-840c-103833c83b2b-kube-api-access-jhllz\") pod \"ee486b7f-543e-4475-840c-103833c83b2b\" (UID: \"ee486b7f-543e-4475-840c-103833c83b2b\") " Nov 03 00:11:17.163560 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.162156 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee486b7f-543e-4475-840c-103833c83b2b-maas-api-openshift-ca" pod "ee486b7f-543e-4475-840c-103833c83b2b" (UID: "ee486b7f-543e-4475-840c-103833c83b2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:11:17.165219 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.165090 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee486b7f-543e-4475-840c-103833c83b2b-maas-api-xks-ca" pod "ee486b7f-543e-4475-840c-103833c83b2b" (UID: "ee486b7f-543e-4475-840c-103833c83b2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:11:17.165219 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.165126 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee486b7f-543e-4475-840c-103833c83b2b-kube-api-access-jhllz" pod "ee486b7f-543e-4475-840c-103833c83b2b" (UID: "ee486b7f-543e-4475-840c-103833c83b2b"). InnerVolumeSpecName "kube-api-access-jhllz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:11:17.263841 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.263321 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee486b7f-543e-4475-840c-103833c83b2b-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:17.263841 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.263354 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhllz\" (UniqueName: \"kubernetes.io/projected/ee486b7f-543e-4475-840c-103833c83b2b-kube-api-access-jhllz\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:17.263841 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.263365 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee486b7f-543e-4475-840c-103833c83b2b-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:17.567285 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.567261 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9948ecf8-wg2ls"] Nov 03 00:11:17.569410 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:11:17.569382 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e916d42_6ee5_41dd_a6c6_c331286e11d3.slice/crio-f19618d4c0f17a914dd277f6b3cae94858bf668c712cac32e61098a285519eaa WatchSource:0}: Error finding container f19618d4c0f17a914dd277f6b3cae94858bf668c712cac32e61098a285519eaa: Status 404 returned error can't find the container with id f19618d4c0f17a914dd277f6b3cae94858bf668c712cac32e61098a285519eaa Nov 03 00:11:17.862953 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.862909 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9948ecf8-wg2ls" event={"ID":"2e916d42-6ee5-41dd-a6c6-c331286e11d3","Type":"ContainerStarted","Data":"7d90f8ca1f452a670dad2428c409185961ba8923a55ef1f7ad390fa65f085a0d"} Nov 03 00:11:17.863465 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.863007 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9948ecf8-wg2ls" event={"ID":"2e916d42-6ee5-41dd-a6c6-c331286e11d3","Type":"ContainerStarted","Data":"f19618d4c0f17a914dd277f6b3cae94858bf668c712cac32e61098a285519eaa"} Nov 03 00:11:17.864708 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.864688 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5616f809-l5dnn" Nov 03 00:11:17.865139 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.864678 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5616f809-l5dnn" event={"ID":"ee486b7f-543e-4475-840c-103833c83b2b","Type":"ContainerDied","Data":"9ca3f7ad4158c582677032fdcaeb4c84c0dc8bb4f8c04c7d3e77e40a50955bd5"} Nov 03 00:11:17.865265 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.865248 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ca3f7ad4158c582677032fdcaeb4c84c0dc8bb4f8c04c7d3e77e40a50955bd5" Nov 03 00:11:17.918521 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.913357 5114 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 03 00:11:17.918521 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.913425 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" podUID="14d58c32-6605-4191-a4cd-3a311461a25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 03 00:11:17.998799 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:17.998738 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9948ecf8-wg2ls" podStartSLOduration=1.998719789 podStartE2EDuration="1.998719789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:11:17.996143387 +0000 UTC m=+4953.349176763" watchObservedRunningTime="2026-11-03 00:11:17.998719789 +0000 UTC m=+4953.351753165" Nov 03 00:11:18.871746 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:18.871644 5114 generic.go:356] "Generic (PLEG): container finished" podID="2e916d42-6ee5-41dd-a6c6-c331286e11d3" containerID="7d90f8ca1f452a670dad2428c409185961ba8923a55ef1f7ad390fa65f085a0d" exitCode=0 Nov 03 00:11:18.872153 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:18.871733 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9948ecf8-wg2ls" event={"ID":"2e916d42-6ee5-41dd-a6c6-c331286e11d3","Type":"ContainerDied","Data":"7d90f8ca1f452a670dad2428c409185961ba8923a55ef1f7ad390fa65f085a0d"} Nov 03 00:11:18.930695 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:18.930658 5114 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 03 00:11:18.930878 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:18.930724 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" podUID="14d58c32-6605-4191-a4cd-3a311461a25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 03 00:11:19.997188 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:19.997155 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:19.998115 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:19.998088 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:19.998845 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:19.998816 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:11:20.029378 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:20.029345 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9948ecf8-wg2ls" Nov 03 00:11:20.093733 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:20.093695 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e916d42-6ee5-41dd-a6c6-c331286e11d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e916d42-6ee5-41dd-a6c6-c331286e11d3-maas-api-xks-ca\") pod \"2e916d42-6ee5-41dd-a6c6-c331286e11d3\" (UID: \"2e916d42-6ee5-41dd-a6c6-c331286e11d3\") " Nov 03 00:11:20.093939 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:20.093810 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e916d42-6ee5-41dd-a6c6-c331286e11d3-kube-api-access-lpfh4\" (UniqueName: \"kubernetes.io/projected/2e916d42-6ee5-41dd-a6c6-c331286e11d3-kube-api-access-lpfh4\") pod \"2e916d42-6ee5-41dd-a6c6-c331286e11d3\" (UID: \"2e916d42-6ee5-41dd-a6c6-c331286e11d3\") " Nov 03 00:11:20.093939 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:20.093871 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e916d42-6ee5-41dd-a6c6-c331286e11d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e916d42-6ee5-41dd-a6c6-c331286e11d3-maas-api-openshift-ca\") pod \"2e916d42-6ee5-41dd-a6c6-c331286e11d3\" (UID: \"2e916d42-6ee5-41dd-a6c6-c331286e11d3\") " Nov 03 00:11:20.094281 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:20.094250 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e916d42-6ee5-41dd-a6c6-c331286e11d3-maas-api-openshift-ca" pod "2e916d42-6ee5-41dd-a6c6-c331286e11d3" (UID: "2e916d42-6ee5-41dd-a6c6-c331286e11d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:11:20.096247 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:20.096225 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e916d42-6ee5-41dd-a6c6-c331286e11d3-kube-api-access-lpfh4" pod "2e916d42-6ee5-41dd-a6c6-c331286e11d3" (UID: "2e916d42-6ee5-41dd-a6c6-c331286e11d3"). InnerVolumeSpecName "kube-api-access-lpfh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:11:20.096351 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:20.096336 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e916d42-6ee5-41dd-a6c6-c331286e11d3-maas-api-xks-ca" pod "2e916d42-6ee5-41dd-a6c6-c331286e11d3" (UID: "2e916d42-6ee5-41dd-a6c6-c331286e11d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:11:20.195703 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:20.195598 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e916d42-6ee5-41dd-a6c6-c331286e11d3-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:20.195703 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:20.195640 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e916d42-6ee5-41dd-a6c6-c331286e11d3-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:20.195703 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:20.195654 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpfh4\" (UniqueName: \"kubernetes.io/projected/2e916d42-6ee5-41dd-a6c6-c331286e11d3-kube-api-access-lpfh4\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:20.883233 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:20.883202 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9948ecf8-wg2ls" Nov 03 00:11:20.883433 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:20.883229 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9948ecf8-wg2ls" event={"ID":"2e916d42-6ee5-41dd-a6c6-c331286e11d3","Type":"ContainerDied","Data":"f19618d4c0f17a914dd277f6b3cae94858bf668c712cac32e61098a285519eaa"} Nov 03 00:11:20.883433 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:20.883276 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f19618d4c0f17a914dd277f6b3cae94858bf668c712cac32e61098a285519eaa" Nov 03 00:11:24.752130 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:24.752038 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d41e706a-sznhb"] Nov 03 00:11:24.752571 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:24.752387 5114 cpu_manager.go:414] "containerMap: removing container" podUID="ee486b7f-543e-4475-840c-103833c83b2b" containerName="revoke-keys" Nov 03 00:11:24.752571 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:24.752398 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee486b7f-543e-4475-840c-103833c83b2b" containerName="revoke-keys" Nov 03 00:11:24.752571 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:24.752409 5114 cpu_manager.go:414] "containerMap: removing container" podUID="2e916d42-6ee5-41dd-a6c6-c331286e11d3" containerName="revoke-keys" Nov 03 00:11:24.752571 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:24.752414 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e916d42-6ee5-41dd-a6c6-c331286e11d3" containerName="revoke-keys" Nov 03 00:11:24.752571 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:24.752507 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee486b7f-543e-4475-840c-103833c83b2b" containerName="revoke-keys" Nov 03 00:11:24.752571 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:24.752515 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e916d42-6ee5-41dd-a6c6-c331286e11d3" containerName="revoke-keys" Nov 03 00:11:24.755988 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:24.755969 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41e706a-sznhb" Nov 03 00:11:24.842535 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:24.842472 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78db8bc7-0c5b-4a63-82ed-b6d8e76ee077-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d41e706a-sznhb\" (UID: \"78db8bc7-0c5b-4a63-82ed-b6d8e76ee077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41e706a-sznhb" Nov 03 00:11:24.842734 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:24.842600 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78db8bc7-0c5b-4a63-82ed-b6d8e76ee077-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d41e706a-sznhb\" (UID: \"78db8bc7-0c5b-4a63-82ed-b6d8e76ee077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41e706a-sznhb" Nov 03 00:11:24.842734 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:24.842666 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjsd8\" (UniqueName: \"kubernetes.io/projected/78db8bc7-0c5b-4a63-82ed-b6d8e76ee077-kube-api-access-gjsd8\") pod \"maas-api-revoke-sub-d41e706a-sznhb\" (UID: \"78db8bc7-0c5b-4a63-82ed-b6d8e76ee077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41e706a-sznhb" Nov 03 00:11:24.860094 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:24.860056 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:11:24.860094 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:24.860062 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 03 00:11:24.862252 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:24.862225 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d41e706a-sznhb"] Nov 03 00:11:24.943314 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:24.943272 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjsd8\" (UniqueName: \"kubernetes.io/projected/78db8bc7-0c5b-4a63-82ed-b6d8e76ee077-kube-api-access-gjsd8\") pod \"maas-api-revoke-sub-d41e706a-sznhb\" (UID: \"78db8bc7-0c5b-4a63-82ed-b6d8e76ee077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41e706a-sznhb" Nov 03 00:11:24.943555 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:24.943331 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78db8bc7-0c5b-4a63-82ed-b6d8e76ee077-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d41e706a-sznhb\" (UID: \"78db8bc7-0c5b-4a63-82ed-b6d8e76ee077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41e706a-sznhb" Nov 03 00:11:24.943555 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:24.943366 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78db8bc7-0c5b-4a63-82ed-b6d8e76ee077-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d41e706a-sznhb\" (UID: \"78db8bc7-0c5b-4a63-82ed-b6d8e76ee077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41e706a-sznhb" Nov 03 00:11:24.944099 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:24.944075 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78db8bc7-0c5b-4a63-82ed-b6d8e76ee077-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d41e706a-sznhb\" (UID: \"78db8bc7-0c5b-4a63-82ed-b6d8e76ee077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41e706a-sznhb" Nov 03 00:11:24.946058 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:24.946035 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78db8bc7-0c5b-4a63-82ed-b6d8e76ee077-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d41e706a-sznhb\" (UID: \"78db8bc7-0c5b-4a63-82ed-b6d8e76ee077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41e706a-sznhb" Nov 03 00:11:25.018887 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:25.018802 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjsd8\" (UniqueName: \"kubernetes.io/projected/78db8bc7-0c5b-4a63-82ed-b6d8e76ee077-kube-api-access-gjsd8\") pod \"maas-api-revoke-sub-d41e706a-sznhb\" (UID: \"78db8bc7-0c5b-4a63-82ed-b6d8e76ee077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41e706a-sznhb" Nov 03 00:11:25.068763 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:25.068703 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41e706a-sznhb" Nov 03 00:11:25.444928 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:25.444899 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d41e706a-sznhb"] Nov 03 00:11:25.447156 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:11:25.447121 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78db8bc7_0c5b_4a63_82ed_b6d8e76ee077.slice/crio-32c456d5519d90f344279ca865b91ffbfc05ee326cc6a88052e611238a806ab2 WatchSource:0}: Error finding container 32c456d5519d90f344279ca865b91ffbfc05ee326cc6a88052e611238a806ab2: Status 404 returned error can't find the container with id 32c456d5519d90f344279ca865b91ffbfc05ee326cc6a88052e611238a806ab2 Nov 03 00:11:25.910767 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:25.910731 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41e706a-sznhb" event={"ID":"78db8bc7-0c5b-4a63-82ed-b6d8e76ee077","Type":"ContainerStarted","Data":"d54e978a1af74f681bf60d8b78df0887a3d4ec6de274d391679fbf6670c101f4"} Nov 03 00:11:25.911173 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:25.910789 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41e706a-sznhb" event={"ID":"78db8bc7-0c5b-4a63-82ed-b6d8e76ee077","Type":"ContainerStarted","Data":"32c456d5519d90f344279ca865b91ffbfc05ee326cc6a88052e611238a806ab2"} Nov 03 00:11:26.027965 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:26.027856 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41e706a-sznhb" podStartSLOduration=2.02783959 podStartE2EDuration="2.02783959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:11:26.025951315 +0000 UTC m=+4961.378984691" watchObservedRunningTime="2026-11-03 00:11:26.02783959 +0000 UTC m=+4961.380872965" Nov 03 00:11:26.917743 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:26.917707 5114 generic.go:356] "Generic (PLEG): container finished" podID="78db8bc7-0c5b-4a63-82ed-b6d8e76ee077" containerID="d54e978a1af74f681bf60d8b78df0887a3d4ec6de274d391679fbf6670c101f4" exitCode=0 Nov 03 00:11:26.918195 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:26.917794 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41e706a-sznhb" event={"ID":"78db8bc7-0c5b-4a63-82ed-b6d8e76ee077","Type":"ContainerDied","Data":"d54e978a1af74f681bf60d8b78df0887a3d4ec6de274d391679fbf6670c101f4"} Nov 03 00:11:28.064189 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:28.064155 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41e706a-sznhb" Nov 03 00:11:28.174399 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:28.174308 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78db8bc7-0c5b-4a63-82ed-b6d8e76ee077-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78db8bc7-0c5b-4a63-82ed-b6d8e76ee077-maas-api-xks-ca\") pod \"78db8bc7-0c5b-4a63-82ed-b6d8e76ee077\" (UID: \"78db8bc7-0c5b-4a63-82ed-b6d8e76ee077\") " Nov 03 00:11:28.174607 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:28.174416 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78db8bc7-0c5b-4a63-82ed-b6d8e76ee077-kube-api-access-gjsd8\" (UniqueName: \"kubernetes.io/projected/78db8bc7-0c5b-4a63-82ed-b6d8e76ee077-kube-api-access-gjsd8\") pod \"78db8bc7-0c5b-4a63-82ed-b6d8e76ee077\" (UID: \"78db8bc7-0c5b-4a63-82ed-b6d8e76ee077\") " Nov 03 00:11:28.174607 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:28.174568 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78db8bc7-0c5b-4a63-82ed-b6d8e76ee077-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78db8bc7-0c5b-4a63-82ed-b6d8e76ee077-maas-api-openshift-ca\") pod \"78db8bc7-0c5b-4a63-82ed-b6d8e76ee077\" (UID: \"78db8bc7-0c5b-4a63-82ed-b6d8e76ee077\") " Nov 03 00:11:28.174911 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:28.174887 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78db8bc7-0c5b-4a63-82ed-b6d8e76ee077-maas-api-openshift-ca" pod "78db8bc7-0c5b-4a63-82ed-b6d8e76ee077" (UID: "78db8bc7-0c5b-4a63-82ed-b6d8e76ee077"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:11:28.176755 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:28.176720 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78db8bc7-0c5b-4a63-82ed-b6d8e76ee077-maas-api-xks-ca" pod "78db8bc7-0c5b-4a63-82ed-b6d8e76ee077" (UID: "78db8bc7-0c5b-4a63-82ed-b6d8e76ee077"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:11:28.176755 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:28.176731 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78db8bc7-0c5b-4a63-82ed-b6d8e76ee077-kube-api-access-gjsd8" pod "78db8bc7-0c5b-4a63-82ed-b6d8e76ee077" (UID: "78db8bc7-0c5b-4a63-82ed-b6d8e76ee077"). InnerVolumeSpecName "kube-api-access-gjsd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:11:28.275700 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:28.275652 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78db8bc7-0c5b-4a63-82ed-b6d8e76ee077-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:28.275700 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:28.275694 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78db8bc7-0c5b-4a63-82ed-b6d8e76ee077-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:28.275700 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:28.275705 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjsd8\" (UniqueName: \"kubernetes.io/projected/78db8bc7-0c5b-4a63-82ed-b6d8e76ee077-kube-api-access-gjsd8\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:28.934858 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:28.934823 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41e706a-sznhb" Nov 03 00:11:28.935084 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:28.934821 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41e706a-sznhb" event={"ID":"78db8bc7-0c5b-4a63-82ed-b6d8e76ee077","Type":"ContainerDied","Data":"32c456d5519d90f344279ca865b91ffbfc05ee326cc6a88052e611238a806ab2"} Nov 03 00:11:28.935084 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:28.934969 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32c456d5519d90f344279ca865b91ffbfc05ee326cc6a88052e611238a806ab2" Nov 03 00:11:30.261207 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:30.261169 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8"] Nov 03 00:11:35.682047 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:35.682004 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c3fafca-mgsqv"] Nov 03 00:11:35.682466 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:35.682380 5114 cpu_manager.go:414] "containerMap: removing container" podUID="78db8bc7-0c5b-4a63-82ed-b6d8e76ee077" containerName="revoke-keys" Nov 03 00:11:35.682466 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:35.682392 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78db8bc7-0c5b-4a63-82ed-b6d8e76ee077" containerName="revoke-keys" Nov 03 00:11:35.682466 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:35.682445 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="78db8bc7-0c5b-4a63-82ed-b6d8e76ee077" containerName="revoke-keys" Nov 03 00:11:35.685827 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:35.685808 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c3fafca-mgsqv" Nov 03 00:11:35.740754 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:35.740712 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:11:35.757429 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:35.757387 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-584kv\" (UniqueName: \"kubernetes.io/projected/be2c30af-068f-4165-9f57-567a24590206-kube-api-access-584kv\") pod \"maas-api-revoke-sub-7c3fafca-mgsqv\" (UID: \"be2c30af-068f-4165-9f57-567a24590206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c3fafca-mgsqv" Nov 03 00:11:35.757607 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:35.757449 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be2c30af-068f-4165-9f57-567a24590206-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c3fafca-mgsqv\" (UID: \"be2c30af-068f-4165-9f57-567a24590206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c3fafca-mgsqv" Nov 03 00:11:35.757674 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:35.757604 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be2c30af-068f-4165-9f57-567a24590206-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c3fafca-mgsqv\" (UID: \"be2c30af-068f-4165-9f57-567a24590206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c3fafca-mgsqv" Nov 03 00:11:35.790463 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:35.790428 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 03 00:11:35.792445 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:35.792418 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c3fafca-mgsqv"] Nov 03 00:11:35.859136 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:35.859092 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-584kv\" (UniqueName: \"kubernetes.io/projected/be2c30af-068f-4165-9f57-567a24590206-kube-api-access-584kv\") pod \"maas-api-revoke-sub-7c3fafca-mgsqv\" (UID: \"be2c30af-068f-4165-9f57-567a24590206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c3fafca-mgsqv" Nov 03 00:11:35.859136 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:35.859137 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be2c30af-068f-4165-9f57-567a24590206-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c3fafca-mgsqv\" (UID: \"be2c30af-068f-4165-9f57-567a24590206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c3fafca-mgsqv" Nov 03 00:11:35.859459 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:35.859173 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be2c30af-068f-4165-9f57-567a24590206-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c3fafca-mgsqv\" (UID: \"be2c30af-068f-4165-9f57-567a24590206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c3fafca-mgsqv" Nov 03 00:11:35.859881 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:35.859862 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be2c30af-068f-4165-9f57-567a24590206-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c3fafca-mgsqv\" (UID: \"be2c30af-068f-4165-9f57-567a24590206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c3fafca-mgsqv" Nov 03 00:11:35.861787 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:35.861765 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be2c30af-068f-4165-9f57-567a24590206-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c3fafca-mgsqv\" (UID: \"be2c30af-068f-4165-9f57-567a24590206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c3fafca-mgsqv" Nov 03 00:11:35.920162 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:35.920118 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-584kv\" (UniqueName: \"kubernetes.io/projected/be2c30af-068f-4165-9f57-567a24590206-kube-api-access-584kv\") pod \"maas-api-revoke-sub-7c3fafca-mgsqv\" (UID: \"be2c30af-068f-4165-9f57-567a24590206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c3fafca-mgsqv" Nov 03 00:11:35.998650 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:35.998555 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c3fafca-mgsqv" Nov 03 00:11:36.374168 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:36.374136 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c3fafca-mgsqv"] Nov 03 00:11:36.376865 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:11:36.376838 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe2c30af_068f_4165_9f57_567a24590206.slice/crio-3a3a5e7a3bad7653cda7ce79d681405d542bc8a16c750cb3d9d316096abea89a WatchSource:0}: Error finding container 3a3a5e7a3bad7653cda7ce79d681405d542bc8a16c750cb3d9d316096abea89a: Status 404 returned error can't find the container with id 3a3a5e7a3bad7653cda7ce79d681405d542bc8a16c750cb3d9d316096abea89a Nov 03 00:11:36.983148 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:36.983058 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c3fafca-mgsqv" event={"ID":"be2c30af-068f-4165-9f57-567a24590206","Type":"ContainerStarted","Data":"0625c8d20f051bf406eb843748601f275e5593d96d69c7d0e173ec773c44447e"} Nov 03 00:11:36.983148 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:36.983113 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c3fafca-mgsqv" event={"ID":"be2c30af-068f-4165-9f57-567a24590206","Type":"ContainerStarted","Data":"3a3a5e7a3bad7653cda7ce79d681405d542bc8a16c750cb3d9d316096abea89a"} Nov 03 00:11:37.098837 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:37.098779 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c3fafca-mgsqv" podStartSLOduration=2.098763603 podStartE2EDuration="2.098763603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:11:37.097559601 +0000 UTC m=+4972.450592977" watchObservedRunningTime="2026-11-03 00:11:37.098763603 +0000 UTC m=+4972.451796956" Nov 03 00:11:37.583792 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:11:37.583700 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:11:37.999206 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:37.999106 5114 generic.go:356] "Generic (PLEG): container finished" podID="be2c30af-068f-4165-9f57-567a24590206" containerID="0625c8d20f051bf406eb843748601f275e5593d96d69c7d0e173ec773c44447e" exitCode=0 Nov 03 00:11:37.999663 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:37.999211 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c3fafca-mgsqv" event={"ID":"be2c30af-068f-4165-9f57-567a24590206","Type":"ContainerDied","Data":"0625c8d20f051bf406eb843748601f275e5593d96d69c7d0e173ec773c44447e"} Nov 03 00:11:39.142362 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:39.142332 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c3fafca-mgsqv" Nov 03 00:11:39.294585 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:39.294548 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be2c30af-068f-4165-9f57-567a24590206-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be2c30af-068f-4165-9f57-567a24590206-maas-api-openshift-ca\") pod \"be2c30af-068f-4165-9f57-567a24590206\" (UID: \"be2c30af-068f-4165-9f57-567a24590206\") " Nov 03 00:11:39.294775 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:39.294625 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be2c30af-068f-4165-9f57-567a24590206-kube-api-access-584kv\" (UniqueName: \"kubernetes.io/projected/be2c30af-068f-4165-9f57-567a24590206-kube-api-access-584kv\") pod \"be2c30af-068f-4165-9f57-567a24590206\" (UID: \"be2c30af-068f-4165-9f57-567a24590206\") " Nov 03 00:11:39.294775 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:39.294753 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be2c30af-068f-4165-9f57-567a24590206-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be2c30af-068f-4165-9f57-567a24590206-maas-api-xks-ca\") pod \"be2c30af-068f-4165-9f57-567a24590206\" (UID: \"be2c30af-068f-4165-9f57-567a24590206\") " Nov 03 00:11:39.295017 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:39.294992 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be2c30af-068f-4165-9f57-567a24590206-maas-api-openshift-ca" pod "be2c30af-068f-4165-9f57-567a24590206" (UID: "be2c30af-068f-4165-9f57-567a24590206"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:11:39.295099 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:39.295081 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be2c30af-068f-4165-9f57-567a24590206-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:39.297056 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:39.297030 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be2c30af-068f-4165-9f57-567a24590206-maas-api-xks-ca" pod "be2c30af-068f-4165-9f57-567a24590206" (UID: "be2c30af-068f-4165-9f57-567a24590206"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:11:39.297150 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:39.297082 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2c30af-068f-4165-9f57-567a24590206-kube-api-access-584kv" pod "be2c30af-068f-4165-9f57-567a24590206" (UID: "be2c30af-068f-4165-9f57-567a24590206"). InnerVolumeSpecName "kube-api-access-584kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:11:39.395775 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:39.395727 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be2c30af-068f-4165-9f57-567a24590206-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:39.395775 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:39.395767 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-584kv\" (UniqueName: \"kubernetes.io/projected/be2c30af-068f-4165-9f57-567a24590206-kube-api-access-584kv\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:40.011175 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:40.011144 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c3fafca-mgsqv" Nov 03 00:11:40.011378 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:40.011146 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c3fafca-mgsqv" event={"ID":"be2c30af-068f-4165-9f57-567a24590206","Type":"ContainerDied","Data":"3a3a5e7a3bad7653cda7ce79d681405d542bc8a16c750cb3d9d316096abea89a"} Nov 03 00:11:40.011378 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:40.011271 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a3a5e7a3bad7653cda7ce79d681405d542bc8a16c750cb3d9d316096abea89a" Nov 03 00:11:40.046192 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:40.046141 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9b63b44-wlvss"] Nov 03 00:11:40.095945 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:40.095910 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9b63b44-wlvss"] Nov 03 00:11:40.792752 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:40.792719 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab305d6b-09c6-4e53-94fa-49e42facf5f4" path="/var/lib/kubelet/pods/ab305d6b-09c6-4e53-94fa-49e42facf5f4/volumes" Nov 03 00:11:45.284344 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:45.284291 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" podUID="82b57f4a-5485-4607-b2e9-1c364f3d4b35" containerName="main" containerID="cri-o://37f20a646fd0fb4ff89621b90666c9f2b4b91426bfc4da53403f1d8f3003dcc6" gracePeriod=45 Nov 03 00:11:45.543129 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:45.543045 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:11:45.658467 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:45.658427 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-tmp-dir\") pod \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\" (UID: \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\") " Nov 03 00:11:45.658719 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:45.658477 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-model-cache\") pod \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\" (UID: \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\") " Nov 03 00:11:45.658719 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:45.658538 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-dshm\" (UniqueName: \"kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-dshm\") pod \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\" (UID: \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\") " Nov 03 00:11:45.658719 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:45.658561 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82b57f4a-5485-4607-b2e9-1c364f3d4b35-tls-certs\" (UniqueName: \"kubernetes.io/secret/82b57f4a-5485-4607-b2e9-1c364f3d4b35-tls-certs\") pod \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\" (UID: \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\") " Nov 03 00:11:45.658719 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:45.658634 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82b57f4a-5485-4607-b2e9-1c364f3d4b35-kube-api-access-lk2dh\" (UniqueName: \"kubernetes.io/projected/82b57f4a-5485-4607-b2e9-1c364f3d4b35-kube-api-access-lk2dh\") pod \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\" (UID: \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\") " Nov 03 00:11:45.658719 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:45.658710 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-home\" (UniqueName: \"kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-home\") pod \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\" (UID: \"82b57f4a-5485-4607-b2e9-1c364f3d4b35\") " Nov 03 00:11:45.660165 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:45.658783 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-tmp-dir" pod "82b57f4a-5485-4607-b2e9-1c364f3d4b35" (UID: "82b57f4a-5485-4607-b2e9-1c364f3d4b35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:11:45.660165 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:45.658794 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-model-cache" pod "82b57f4a-5485-4607-b2e9-1c364f3d4b35" (UID: "82b57f4a-5485-4607-b2e9-1c364f3d4b35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:11:45.660165 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:45.659233 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:45.660165 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:45.659256 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:45.660165 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:45.659416 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-home" pod "82b57f4a-5485-4607-b2e9-1c364f3d4b35" (UID: "82b57f4a-5485-4607-b2e9-1c364f3d4b35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:11:45.662266 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:45.661998 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-dshm" pod "82b57f4a-5485-4607-b2e9-1c364f3d4b35" (UID: "82b57f4a-5485-4607-b2e9-1c364f3d4b35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:11:45.662266 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:45.662003 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b57f4a-5485-4607-b2e9-1c364f3d4b35-kube-api-access-lk2dh" pod "82b57f4a-5485-4607-b2e9-1c364f3d4b35" (UID: "82b57f4a-5485-4607-b2e9-1c364f3d4b35"). InnerVolumeSpecName "kube-api-access-lk2dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:11:45.662588 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:45.662562 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82b57f4a-5485-4607-b2e9-1c364f3d4b35-tls-certs" pod "82b57f4a-5485-4607-b2e9-1c364f3d4b35" (UID: "82b57f4a-5485-4607-b2e9-1c364f3d4b35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:11:45.760746 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:45.760697 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:45.760746 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:45.760739 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82b57f4a-5485-4607-b2e9-1c364f3d4b35-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:45.760999 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:45.760755 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk2dh\" (UniqueName: \"kubernetes.io/projected/82b57f4a-5485-4607-b2e9-1c364f3d4b35-kube-api-access-lk2dh\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:45.760999 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:45.760770 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82b57f4a-5485-4607-b2e9-1c364f3d4b35-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:46.042666 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:46.042627 5114 generic.go:356] "Generic (PLEG): container finished" podID="82b57f4a-5485-4607-b2e9-1c364f3d4b35" containerID="37f20a646fd0fb4ff89621b90666c9f2b4b91426bfc4da53403f1d8f3003dcc6" exitCode=0 Nov 03 00:11:46.042872 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:46.042683 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" Nov 03 00:11:46.042872 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:46.042721 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" event={"ID":"82b57f4a-5485-4607-b2e9-1c364f3d4b35","Type":"ContainerDied","Data":"37f20a646fd0fb4ff89621b90666c9f2b4b91426bfc4da53403f1d8f3003dcc6"} Nov 03 00:11:46.042872 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:46.042764 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8" event={"ID":"82b57f4a-5485-4607-b2e9-1c364f3d4b35","Type":"ContainerDied","Data":"958e48d8b2b40afc5c6ead63356dec3b36fdcba802c0a7a1ae25849e0f120e75"} Nov 03 00:11:46.042872 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:46.042788 5114 scope.go:122] "RemoveContainer" containerID="37f20a646fd0fb4ff89621b90666c9f2b4b91426bfc4da53403f1d8f3003dcc6" Nov 03 00:11:46.052640 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:46.052617 5114 scope.go:122] "RemoveContainer" containerID="37f20a646fd0fb4ff89621b90666c9f2b4b91426bfc4da53403f1d8f3003dcc6" Nov 03 00:11:46.052974 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:11:46.052944 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37f20a646fd0fb4ff89621b90666c9f2b4b91426bfc4da53403f1d8f3003dcc6\": container with ID starting with 37f20a646fd0fb4ff89621b90666c9f2b4b91426bfc4da53403f1d8f3003dcc6 not found: ID does not exist" containerID="37f20a646fd0fb4ff89621b90666c9f2b4b91426bfc4da53403f1d8f3003dcc6" Nov 03 00:11:46.053053 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:46.052986 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37f20a646fd0fb4ff89621b90666c9f2b4b91426bfc4da53403f1d8f3003dcc6"} err="failed to get container status \"37f20a646fd0fb4ff89621b90666c9f2b4b91426bfc4da53403f1d8f3003dcc6\": rpc error: code = NotFound desc = could not find container \"37f20a646fd0fb4ff89621b90666c9f2b4b91426bfc4da53403f1d8f3003dcc6\": container with ID starting with 37f20a646fd0fb4ff89621b90666c9f2b4b91426bfc4da53403f1d8f3003dcc6 not found: ID does not exist" Nov 03 00:11:46.206201 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:46.206164 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8"] Nov 03 00:11:46.210553 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:46.210528 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9b6c9e/rate-test-model-9b6c9e-kserve-84b9cc877-67jv8"] Nov 03 00:11:46.793233 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:46.793192 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82b57f4a-5485-4607-b2e9-1c364f3d4b35" path="/var/lib/kubelet/pods/82b57f4a-5485-4607-b2e9-1c364f3d4b35/volumes" Nov 03 00:11:46.983673 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:46.983641 5114 scope.go:122] "RemoveContainer" containerID="9abe1528fadb543f4d4138a34232b3d0f42d1f7d0bf33f61a1f55ade095778c3" Nov 03 00:11:46.992721 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:46.992696 5114 scope.go:122] "RemoveContainer" containerID="dd30f00b923386937322e4a60dce2d7b3013f33e5d930ac0355414028a39270c" Nov 03 00:11:47.001322 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:47.001301 5114 scope.go:122] "RemoveContainer" containerID="8d6b000a29476736c4086bc524efcd7beac6114e46c34ad7242b595340e23143" Nov 03 00:11:47.009709 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:47.009687 5114 scope.go:122] "RemoveContainer" containerID="9673699b578a19dd536ca8c702c4dc749a0cf2c11b71be546dbd578a67d70369" Nov 03 00:11:57.047677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:57.047635 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd12d8e-sqz2s"] Nov 03 00:11:57.096775 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:57.096730 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd12d8e-sqz2s"] Nov 03 00:11:58.792743 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:11:58.792700 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4b5c598-38d9-445d-97a0-724c1a83ef02" path="/var/lib/kubelet/pods/a4b5c598-38d9-445d-97a0-724c1a83ef02/volumes" Nov 03 00:12:07.584352 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:12:07.584310 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:12:09.078143 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.078098 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96"] Nov 03 00:12:09.078660 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.078449 5114 cpu_manager.go:414] "containerMap: removing container" podUID="be2c30af-068f-4165-9f57-567a24590206" containerName="revoke-keys" Nov 03 00:12:09.078660 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.078461 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be2c30af-068f-4165-9f57-567a24590206" containerName="revoke-keys" Nov 03 00:12:09.078660 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.078504 5114 cpu_manager.go:414] "containerMap: removing container" podUID="82b57f4a-5485-4607-b2e9-1c364f3d4b35" containerName="main" Nov 03 00:12:09.078660 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.078511 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82b57f4a-5485-4607-b2e9-1c364f3d4b35" containerName="main" Nov 03 00:12:09.078660 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.078602 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="be2c30af-068f-4165-9f57-567a24590206" containerName="revoke-keys" Nov 03 00:12:09.078660 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.078617 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="82b57f4a-5485-4607-b2e9-1c364f3d4b35" containerName="main" Nov 03 00:12:09.081858 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.081838 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:12:09.137647 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.137475 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-3bee1b\"/\"openshift-service-ca.crt\"" Nov 03 00:12:09.151497 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.151451 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-3bee1b\"/\"kube-root-ca.crt\"" Nov 03 00:12:09.179461 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.179420 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-tmp-dir\") pod \"sub-test-model-3bee1b-kserve-b857999cc-6dq96\" (UID: \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:12:09.179722 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.179498 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lstr2\" (UniqueName: \"kubernetes.io/projected/ef6a64ff-b843-4eae-8af4-49b999bf7e15-kube-api-access-lstr2\") pod \"sub-test-model-3bee1b-kserve-b857999cc-6dq96\" (UID: \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:12:09.179722 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.179597 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-dshm\") pod \"sub-test-model-3bee1b-kserve-b857999cc-6dq96\" (UID: \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:12:09.179722 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.179666 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-model-cache\") pod \"sub-test-model-3bee1b-kserve-b857999cc-6dq96\" (UID: \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:12:09.179880 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.179751 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-home\") pod \"sub-test-model-3bee1b-kserve-b857999cc-6dq96\" (UID: \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:12:09.179880 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.179790 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef6a64ff-b843-4eae-8af4-49b999bf7e15-tls-certs\") pod \"sub-test-model-3bee1b-kserve-b857999cc-6dq96\" (UID: \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:12:09.183796 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.183773 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-3bee1b\"/\"default-dockercfg-j75g4\"" Nov 03 00:12:09.183875 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.183854 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-3bee1b\"/\"sub-test-model-3bee1b-kserve-self-signed-certs\"" Nov 03 00:12:09.185880 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.185857 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96"] Nov 03 00:12:09.281254 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.281216 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-home\") pod \"sub-test-model-3bee1b-kserve-b857999cc-6dq96\" (UID: \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:12:09.281254 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.281259 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef6a64ff-b843-4eae-8af4-49b999bf7e15-tls-certs\") pod \"sub-test-model-3bee1b-kserve-b857999cc-6dq96\" (UID: \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:12:09.281575 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.281315 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-tmp-dir\") pod \"sub-test-model-3bee1b-kserve-b857999cc-6dq96\" (UID: \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:12:09.281575 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.281353 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lstr2\" (UniqueName: \"kubernetes.io/projected/ef6a64ff-b843-4eae-8af4-49b999bf7e15-kube-api-access-lstr2\") pod \"sub-test-model-3bee1b-kserve-b857999cc-6dq96\" (UID: \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:12:09.281575 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.281386 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-dshm\") pod \"sub-test-model-3bee1b-kserve-b857999cc-6dq96\" (UID: \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:12:09.281575 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.281423 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-model-cache\") pod \"sub-test-model-3bee1b-kserve-b857999cc-6dq96\" (UID: \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:12:09.281814 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.281735 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-tmp-dir\") pod \"sub-test-model-3bee1b-kserve-b857999cc-6dq96\" (UID: \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:12:09.281814 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.281724 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-home\") pod \"sub-test-model-3bee1b-kserve-b857999cc-6dq96\" (UID: \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:12:09.281814 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.281783 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-model-cache\") pod \"sub-test-model-3bee1b-kserve-b857999cc-6dq96\" (UID: \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:12:09.283948 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.283914 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-dshm\") pod \"sub-test-model-3bee1b-kserve-b857999cc-6dq96\" (UID: \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:12:09.284219 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.284200 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef6a64ff-b843-4eae-8af4-49b999bf7e15-tls-certs\") pod \"sub-test-model-3bee1b-kserve-b857999cc-6dq96\" (UID: \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:12:09.344923 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.344851 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lstr2\" (UniqueName: \"kubernetes.io/projected/ef6a64ff-b843-4eae-8af4-49b999bf7e15-kube-api-access-lstr2\") pod \"sub-test-model-3bee1b-kserve-b857999cc-6dq96\" (UID: \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\") " pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:12:09.394389 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.394352 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:12:09.787620 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:09.787569 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96"] Nov 03 00:12:09.789330 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:12:09.789300 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef6a64ff_b843_4eae_8af4_49b999bf7e15.slice/crio-afdb0df2ab51029eb8d0e596475a02aa09b24bfb7f973b8860c41b930817a561 WatchSource:0}: Error finding container afdb0df2ab51029eb8d0e596475a02aa09b24bfb7f973b8860c41b930817a561: Status 404 returned error can't find the container with id afdb0df2ab51029eb8d0e596475a02aa09b24bfb7f973b8860c41b930817a561 Nov 03 00:12:10.170780 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:10.170684 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" event={"ID":"ef6a64ff-b843-4eae-8af4-49b999bf7e15","Type":"ContainerStarted","Data":"1a38a9eaf4c8514070f7f4dffb7c21f82385d33b70c4adddf41e7e04acce42af"} Nov 03 00:12:10.170780 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:10.170738 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" event={"ID":"ef6a64ff-b843-4eae-8af4-49b999bf7e15","Type":"ContainerStarted","Data":"afdb0df2ab51029eb8d0e596475a02aa09b24bfb7f973b8860c41b930817a561"} Nov 03 00:12:10.286881 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:10.286818 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" podStartSLOduration=1.286802644 podStartE2EDuration="1.286802644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:12:10.286051491 +0000 UTC m=+5005.639084868" watchObservedRunningTime="2026-11-03 00:12:10.286802644 +0000 UTC m=+5005.639836058" Nov 03 00:12:10.394589 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:10.394553 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:12:11.541390 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:11.541348 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn"] Nov 03 00:12:19.395421 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:19.395380 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:12:19.408830 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:19.408798 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:12:20.235208 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:20.235167 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:12:26.562461 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:26.562419 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" podUID="aa48f41c-68f1-4f33-838b-33f0a1361dd3" containerName="main" containerID="cri-o://fae0955ff4a7bb280b991ed4133bef4782456084f3e9021b444f1511c9229d80" gracePeriod=45 Nov 03 00:12:26.818758 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:26.818685 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:12:26.958551 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:26.958510 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-model-cache\") pod \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\" (UID: \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\") " Nov 03 00:12:26.958758 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:26.958569 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-dshm\") pod \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\" (UID: \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\") " Nov 03 00:12:26.958758 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:26.958604 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-home\" (UniqueName: \"kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-home\") pod \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\" (UID: \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\") " Nov 03 00:12:26.958758 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:26.958725 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa48f41c-68f1-4f33-838b-33f0a1361dd3-tls-certs\" (UniqueName: \"kubernetes.io/secret/aa48f41c-68f1-4f33-838b-33f0a1361dd3-tls-certs\") pod \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\" (UID: \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\") " Nov 03 00:12:26.958939 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:26.958768 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa48f41c-68f1-4f33-838b-33f0a1361dd3-kube-api-access-9f2sc\" (UniqueName: \"kubernetes.io/projected/aa48f41c-68f1-4f33-838b-33f0a1361dd3-kube-api-access-9f2sc\") pod \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\" (UID: \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\") " Nov 03 00:12:26.958939 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:26.958805 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-tmp-dir\") pod \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\" (UID: \"aa48f41c-68f1-4f33-838b-33f0a1361dd3\") " Nov 03 00:12:26.958939 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:26.958813 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-model-cache" pod "aa48f41c-68f1-4f33-838b-33f0a1361dd3" (UID: "aa48f41c-68f1-4f33-838b-33f0a1361dd3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:12:26.958939 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:26.958902 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-home" pod "aa48f41c-68f1-4f33-838b-33f0a1361dd3" (UID: "aa48f41c-68f1-4f33-838b-33f0a1361dd3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:12:26.959161 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:26.959124 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-tmp-dir" pod "aa48f41c-68f1-4f33-838b-33f0a1361dd3" (UID: "aa48f41c-68f1-4f33-838b-33f0a1361dd3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:12:26.959228 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:26.959179 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:12:26.959228 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:26.959194 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:12:26.961190 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:26.961160 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa48f41c-68f1-4f33-838b-33f0a1361dd3-kube-api-access-9f2sc" pod "aa48f41c-68f1-4f33-838b-33f0a1361dd3" (UID: "aa48f41c-68f1-4f33-838b-33f0a1361dd3"). InnerVolumeSpecName "kube-api-access-9f2sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:12:26.961581 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:26.961545 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-dshm" pod "aa48f41c-68f1-4f33-838b-33f0a1361dd3" (UID: "aa48f41c-68f1-4f33-838b-33f0a1361dd3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:12:26.961581 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:26.961552 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa48f41c-68f1-4f33-838b-33f0a1361dd3-tls-certs" pod "aa48f41c-68f1-4f33-838b-33f0a1361dd3" (UID: "aa48f41c-68f1-4f33-838b-33f0a1361dd3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:12:27.060373 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:27.060330 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:12:27.060373 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:27.060382 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa48f41c-68f1-4f33-838b-33f0a1361dd3-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:12:27.060634 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:27.060392 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f2sc\" (UniqueName: \"kubernetes.io/projected/aa48f41c-68f1-4f33-838b-33f0a1361dd3-kube-api-access-9f2sc\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:12:27.060634 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:27.060403 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa48f41c-68f1-4f33-838b-33f0a1361dd3-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:12:27.260813 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:27.260717 5114 generic.go:356] "Generic (PLEG): container finished" podID="aa48f41c-68f1-4f33-838b-33f0a1361dd3" containerID="fae0955ff4a7bb280b991ed4133bef4782456084f3e9021b444f1511c9229d80" exitCode=0 Nov 03 00:12:27.260813 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:27.260759 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" Nov 03 00:12:27.260813 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:27.260791 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" event={"ID":"aa48f41c-68f1-4f33-838b-33f0a1361dd3","Type":"ContainerDied","Data":"fae0955ff4a7bb280b991ed4133bef4782456084f3e9021b444f1511c9229d80"} Nov 03 00:12:27.261112 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:27.260845 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn" event={"ID":"aa48f41c-68f1-4f33-838b-33f0a1361dd3","Type":"ContainerDied","Data":"75a94e269951bf1288efb548135177d2553f38fe66ef8c174fa6c28b772c6918"} Nov 03 00:12:27.261112 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:27.260869 5114 scope.go:122] "RemoveContainer" containerID="fae0955ff4a7bb280b991ed4133bef4782456084f3e9021b444f1511c9229d80" Nov 03 00:12:27.271053 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:27.271031 5114 scope.go:122] "RemoveContainer" containerID="fae0955ff4a7bb280b991ed4133bef4782456084f3e9021b444f1511c9229d80" Nov 03 00:12:27.271392 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:12:27.271374 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fae0955ff4a7bb280b991ed4133bef4782456084f3e9021b444f1511c9229d80\": container with ID starting with fae0955ff4a7bb280b991ed4133bef4782456084f3e9021b444f1511c9229d80 not found: ID does not exist" containerID="fae0955ff4a7bb280b991ed4133bef4782456084f3e9021b444f1511c9229d80" Nov 03 00:12:27.271453 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:27.271401 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fae0955ff4a7bb280b991ed4133bef4782456084f3e9021b444f1511c9229d80"} err="failed to get container status \"fae0955ff4a7bb280b991ed4133bef4782456084f3e9021b444f1511c9229d80\": rpc error: code = NotFound desc = could not find container \"fae0955ff4a7bb280b991ed4133bef4782456084f3e9021b444f1511c9229d80\": container with ID starting with fae0955ff4a7bb280b991ed4133bef4782456084f3e9021b444f1511c9229d80 not found: ID does not exist" Nov 03 00:12:27.424648 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:27.424613 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn"] Nov 03 00:12:27.428075 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:27.428044 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-980b82/sub-test-model-980b82-kserve-5968685f87-r92jn"] Nov 03 00:12:28.793154 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:28.793119 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa48f41c-68f1-4f33-838b-33f0a1361dd3" path="/var/lib/kubelet/pods/aa48f41c-68f1-4f33-838b-33f0a1361dd3/volumes" Nov 03 00:12:37.583637 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:12:37.583600 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:12:39.990243 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:39.990179 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx"] Nov 03 00:12:47.055001 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:47.054967 5114 scope.go:122] "RemoveContainer" containerID="d5678b2acc5340e863ce856a37b59ea6bd6e7b93a49cd473c080e927b25a89d5" Nov 03 00:12:55.010961 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.010844 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" podUID="f9a3771d-32a1-42fc-a172-007488fa523d" containerName="main" containerID="cri-o://828022ce29a8f0a39c061cd5ccd992966b4579c826354b2a31cf84c70eb82770" gracePeriod=45 Nov 03 00:12:55.256453 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.256421 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:12:55.337126 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.337094 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-tmp-dir\") pod \"f9a3771d-32a1-42fc-a172-007488fa523d\" (UID: \"f9a3771d-32a1-42fc-a172-007488fa523d\") " Nov 03 00:12:55.337342 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.337158 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-dshm\") pod \"f9a3771d-32a1-42fc-a172-007488fa523d\" (UID: \"f9a3771d-32a1-42fc-a172-007488fa523d\") " Nov 03 00:12:55.337342 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.337194 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-model-cache\") pod \"f9a3771d-32a1-42fc-a172-007488fa523d\" (UID: \"f9a3771d-32a1-42fc-a172-007488fa523d\") " Nov 03 00:12:55.337342 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.337214 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9a3771d-32a1-42fc-a172-007488fa523d-kube-api-access-89bfr\" (UniqueName: \"kubernetes.io/projected/f9a3771d-32a1-42fc-a172-007488fa523d-kube-api-access-89bfr\") pod \"f9a3771d-32a1-42fc-a172-007488fa523d\" (UID: \"f9a3771d-32a1-42fc-a172-007488fa523d\") " Nov 03 00:12:55.337342 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.337263 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9a3771d-32a1-42fc-a172-007488fa523d-tls-certs\" (UniqueName: \"kubernetes.io/secret/f9a3771d-32a1-42fc-a172-007488fa523d-tls-certs\") pod \"f9a3771d-32a1-42fc-a172-007488fa523d\" (UID: \"f9a3771d-32a1-42fc-a172-007488fa523d\") " Nov 03 00:12:55.337342 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.337283 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-home\" (UniqueName: \"kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-home\") pod \"f9a3771d-32a1-42fc-a172-007488fa523d\" (UID: \"f9a3771d-32a1-42fc-a172-007488fa523d\") " Nov 03 00:12:55.337680 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.337447 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-tmp-dir" pod "f9a3771d-32a1-42fc-a172-007488fa523d" (UID: "f9a3771d-32a1-42fc-a172-007488fa523d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:12:55.337680 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.337505 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-model-cache" pod "f9a3771d-32a1-42fc-a172-007488fa523d" (UID: "f9a3771d-32a1-42fc-a172-007488fa523d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:12:55.337680 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.337616 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-home" pod "f9a3771d-32a1-42fc-a172-007488fa523d" (UID: "f9a3771d-32a1-42fc-a172-007488fa523d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:12:55.337680 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.337639 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:12:55.337680 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.337662 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:12:55.339647 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.339619 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-dshm" pod "f9a3771d-32a1-42fc-a172-007488fa523d" (UID: "f9a3771d-32a1-42fc-a172-007488fa523d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:12:55.339647 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.339635 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9a3771d-32a1-42fc-a172-007488fa523d-tls-certs" pod "f9a3771d-32a1-42fc-a172-007488fa523d" (UID: "f9a3771d-32a1-42fc-a172-007488fa523d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:12:55.339810 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.339647 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a3771d-32a1-42fc-a172-007488fa523d-kube-api-access-89bfr" pod "f9a3771d-32a1-42fc-a172-007488fa523d" (UID: "f9a3771d-32a1-42fc-a172-007488fa523d"). InnerVolumeSpecName "kube-api-access-89bfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:12:55.411872 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.411829 5114 generic.go:356] "Generic (PLEG): container finished" podID="f9a3771d-32a1-42fc-a172-007488fa523d" containerID="828022ce29a8f0a39c061cd5ccd992966b4579c826354b2a31cf84c70eb82770" exitCode=0 Nov 03 00:12:55.412111 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.411882 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" Nov 03 00:12:55.412111 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.411884 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" event={"ID":"f9a3771d-32a1-42fc-a172-007488fa523d","Type":"ContainerDied","Data":"828022ce29a8f0a39c061cd5ccd992966b4579c826354b2a31cf84c70eb82770"} Nov 03 00:12:55.412111 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.411939 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx" event={"ID":"f9a3771d-32a1-42fc-a172-007488fa523d","Type":"ContainerDied","Data":"003f0357327888462f6f70df660b48c3500d7f712da491a0004116e51ceab894"} Nov 03 00:12:55.412111 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.411964 5114 scope.go:122] "RemoveContainer" containerID="828022ce29a8f0a39c061cd5ccd992966b4579c826354b2a31cf84c70eb82770" Nov 03 00:12:55.421884 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.421858 5114 scope.go:122] "RemoveContainer" containerID="828022ce29a8f0a39c061cd5ccd992966b4579c826354b2a31cf84c70eb82770" Nov 03 00:12:55.422217 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:12:55.422197 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828022ce29a8f0a39c061cd5ccd992966b4579c826354b2a31cf84c70eb82770\": container with ID starting with 828022ce29a8f0a39c061cd5ccd992966b4579c826354b2a31cf84c70eb82770 not found: ID does not exist" containerID="828022ce29a8f0a39c061cd5ccd992966b4579c826354b2a31cf84c70eb82770" Nov 03 00:12:55.422298 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.422227 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828022ce29a8f0a39c061cd5ccd992966b4579c826354b2a31cf84c70eb82770"} err="failed to get container status \"828022ce29a8f0a39c061cd5ccd992966b4579c826354b2a31cf84c70eb82770\": rpc error: code = NotFound desc = could not find container \"828022ce29a8f0a39c061cd5ccd992966b4579c826354b2a31cf84c70eb82770\": container with ID starting with 828022ce29a8f0a39c061cd5ccd992966b4579c826354b2a31cf84c70eb82770 not found: ID does not exist" Nov 03 00:12:55.438188 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.438152 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9a3771d-32a1-42fc-a172-007488fa523d-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:12:55.438188 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.438183 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:12:55.438188 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.438193 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9a3771d-32a1-42fc-a172-007488fa523d-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:12:55.438440 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.438202 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89bfr\" (UniqueName: \"kubernetes.io/projected/f9a3771d-32a1-42fc-a172-007488fa523d-kube-api-access-89bfr\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:12:55.576903 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.576865 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx"] Nov 03 00:12:55.580986 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:55.580942 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8d3cb/sub-test-model-b8d3cb-kserve-b87cb8d69-zhsrx"] Nov 03 00:12:56.793471 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:12:56.793433 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9a3771d-32a1-42fc-a172-007488fa523d" path="/var/lib/kubelet/pods/f9a3771d-32a1-42fc-a172-007488fa523d/volumes" Nov 03 00:13:07.583357 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:13:07.583316 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:13:19.566805 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.566755 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m"] Nov 03 00:13:19.567385 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.567097 5114 cpu_manager.go:414] "containerMap: removing container" podUID="aa48f41c-68f1-4f33-838b-33f0a1361dd3" containerName="main" Nov 03 00:13:19.567385 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.567108 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa48f41c-68f1-4f33-838b-33f0a1361dd3" containerName="main" Nov 03 00:13:19.567385 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.567127 5114 cpu_manager.go:414] "containerMap: removing container" podUID="f9a3771d-32a1-42fc-a172-007488fa523d" containerName="main" Nov 03 00:13:19.567385 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.567133 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9a3771d-32a1-42fc-a172-007488fa523d" containerName="main" Nov 03 00:13:19.567385 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.567187 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa48f41c-68f1-4f33-838b-33f0a1361dd3" containerName="main" Nov 03 00:13:19.567385 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.567198 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9a3771d-32a1-42fc-a172-007488fa523d" containerName="main" Nov 03 00:13:19.570401 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.570383 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:13:19.672175 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.672137 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-9b6c9e\"/\"default-dockercfg-w69v5\"" Nov 03 00:13:19.672175 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.672142 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-9b6c9e\"/\"kube-root-ca.crt\"" Nov 03 00:13:19.672175 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.672169 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-9b6c9e\"/\"openshift-service-ca.crt\"" Nov 03 00:13:19.672461 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.672327 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-9b6c9e\"/\"sub-test-model-9b6c9e-kserve-self-signed-certs\"" Nov 03 00:13:19.674152 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.674127 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m"] Nov 03 00:13:19.682745 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.682700 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-model-cache\") pod \"sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m\" (UID: \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:13:19.682899 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.682758 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdvn8\" (UniqueName: \"kubernetes.io/projected/3e31a11c-4ea9-40c5-b4b8-edc638c18523-kube-api-access-wdvn8\") pod \"sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m\" (UID: \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:13:19.682899 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.682883 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-dshm\") pod \"sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m\" (UID: \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:13:19.682998 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.682923 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-home\") pod \"sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m\" (UID: \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:13:19.683076 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.683055 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-tmp-dir\") pod \"sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m\" (UID: \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:13:19.683129 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.683116 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e31a11c-4ea9-40c5-b4b8-edc638c18523-tls-certs\") pod \"sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m\" (UID: \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:13:19.784467 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.784422 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-home\") pod \"sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m\" (UID: \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:13:19.784721 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.784533 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-tmp-dir\") pod \"sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m\" (UID: \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:13:19.784721 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.784583 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e31a11c-4ea9-40c5-b4b8-edc638c18523-tls-certs\") pod \"sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m\" (UID: \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:13:19.784721 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.784630 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-model-cache\") pod \"sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m\" (UID: \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:13:19.784721 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.784667 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdvn8\" (UniqueName: \"kubernetes.io/projected/3e31a11c-4ea9-40c5-b4b8-edc638c18523-kube-api-access-wdvn8\") pod \"sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m\" (UID: \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:13:19.784977 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.784724 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-dshm\") pod \"sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m\" (UID: \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:13:19.785459 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.784971 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-home\") pod \"sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m\" (UID: \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:13:19.785459 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.785020 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-tmp-dir\") pod \"sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m\" (UID: \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:13:19.785459 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.785130 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-model-cache\") pod \"sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m\" (UID: \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:13:19.787204 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.787182 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-dshm\") pod \"sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m\" (UID: \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:13:19.787653 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.787630 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e31a11c-4ea9-40c5-b4b8-edc638c18523-tls-certs\") pod \"sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m\" (UID: \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:13:19.845784 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.845685 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdvn8\" (UniqueName: \"kubernetes.io/projected/3e31a11c-4ea9-40c5-b4b8-edc638c18523-kube-api-access-wdvn8\") pod \"sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m\" (UID: \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\") " pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:13:19.882467 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:19.882424 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:13:20.267096 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:20.266996 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m"] Nov 03 00:13:20.270625 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:13:20.270582 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e31a11c_4ea9_40c5_b4b8_edc638c18523.slice/crio-37a0656a7703ed6e29ffb8f709094234daec6a0440d6169f04e551066a8c39ff WatchSource:0}: Error finding container 37a0656a7703ed6e29ffb8f709094234daec6a0440d6169f04e551066a8c39ff: Status 404 returned error can't find the container with id 37a0656a7703ed6e29ffb8f709094234daec6a0440d6169f04e551066a8c39ff Nov 03 00:13:20.536287 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:20.536252 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" event={"ID":"3e31a11c-4ea9-40c5-b4b8-edc638c18523","Type":"ContainerStarted","Data":"cdb74d2e9f206f09f05b27981b48f48cacec2bde59f40c78bf0811443d04f1ee"} Nov 03 00:13:20.536509 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:20.536299 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" event={"ID":"3e31a11c-4ea9-40c5-b4b8-edc638c18523","Type":"ContainerStarted","Data":"37a0656a7703ed6e29ffb8f709094234daec6a0440d6169f04e551066a8c39ff"} Nov 03 00:13:20.653910 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:20.653845 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" podStartSLOduration=1.653829618 podStartE2EDuration="1.653829618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:20.653547386 +0000 UTC m=+5076.006580762" watchObservedRunningTime="2026-11-03 00:13:20.653829618 +0000 UTC m=+5076.006862987" Nov 03 00:13:20.883646 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:20.883544 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:13:29.883194 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:29.883147 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:13:29.896061 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:29.896029 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:13:30.601003 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:30.600972 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:13:37.583525 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:13:37.583461 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:13:47.081821 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:13:47.079977 5114 scope.go:122] "RemoveContainer" containerID="b3e2972caf4353ad746df73eed712002b4cc57dd1380049ca70f2d1b0316b82d" Nov 03 00:14:04.126833 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:04.126787 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v"] Nov 03 00:14:04.127402 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:04.127025 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" podUID="14d58c32-6605-4191-a4cd-3a311461a25a" containerName="istio-proxy" containerID="cri-o://b11e70f7184e27f11fb038748dd53f54b6e19253b2b5e34b976f8a06b8df3c0a" gracePeriod=30 Nov 03 00:14:05.000792 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:05.000753 5114 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:14:05.001025 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:05.000818 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" podUID="14d58c32-6605-4191-a4cd-3a311461a25a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:14:05.181496 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:05.181456 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41df62c1-9ssfb"] Nov 03 00:14:05.184971 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:05.184943 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41df62c1-9ssfb" Nov 03 00:14:05.236993 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:05.236944 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:14:05.237200 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:05.237055 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 03 00:14:05.243838 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:05.243799 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2049967-6eed-4548-b61a-ea43e34024da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41df62c1-9ssfb\" (UID: \"d2049967-6eed-4548-b61a-ea43e34024da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41df62c1-9ssfb" Nov 03 00:14:05.244040 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:05.243862 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2049967-6eed-4548-b61a-ea43e34024da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41df62c1-9ssfb\" (UID: \"d2049967-6eed-4548-b61a-ea43e34024da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41df62c1-9ssfb" Nov 03 00:14:05.244040 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:05.243948 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8798\" (UniqueName: \"kubernetes.io/projected/d2049967-6eed-4548-b61a-ea43e34024da-kube-api-access-j8798\") pod \"maas-api-revoke-sub-41df62c1-9ssfb\" (UID: \"d2049967-6eed-4548-b61a-ea43e34024da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41df62c1-9ssfb" Nov 03 00:14:05.288206 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:05.288176 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41df62c1-9ssfb"] Nov 03 00:14:05.344851 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:05.344815 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2049967-6eed-4548-b61a-ea43e34024da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41df62c1-9ssfb\" (UID: \"d2049967-6eed-4548-b61a-ea43e34024da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41df62c1-9ssfb" Nov 03 00:14:05.345063 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:05.344868 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2049967-6eed-4548-b61a-ea43e34024da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41df62c1-9ssfb\" (UID: \"d2049967-6eed-4548-b61a-ea43e34024da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41df62c1-9ssfb" Nov 03 00:14:05.345063 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:05.344893 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8798\" (UniqueName: \"kubernetes.io/projected/d2049967-6eed-4548-b61a-ea43e34024da-kube-api-access-j8798\") pod \"maas-api-revoke-sub-41df62c1-9ssfb\" (UID: \"d2049967-6eed-4548-b61a-ea43e34024da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41df62c1-9ssfb" Nov 03 00:14:05.345593 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:05.345567 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2049967-6eed-4548-b61a-ea43e34024da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41df62c1-9ssfb\" (UID: \"d2049967-6eed-4548-b61a-ea43e34024da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41df62c1-9ssfb" Nov 03 00:14:05.347690 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:05.347663 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2049967-6eed-4548-b61a-ea43e34024da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41df62c1-9ssfb\" (UID: \"d2049967-6eed-4548-b61a-ea43e34024da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41df62c1-9ssfb" Nov 03 00:14:05.405431 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:05.405389 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8798\" (UniqueName: \"kubernetes.io/projected/d2049967-6eed-4548-b61a-ea43e34024da-kube-api-access-j8798\") pod \"maas-api-revoke-sub-41df62c1-9ssfb\" (UID: \"d2049967-6eed-4548-b61a-ea43e34024da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41df62c1-9ssfb" Nov 03 00:14:05.498779 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:05.498736 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41df62c1-9ssfb" Nov 03 00:14:05.870722 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:05.870691 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41df62c1-9ssfb"] Nov 03 00:14:05.872933 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:14:05.872901 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2049967_6eed_4548_b61a_ea43e34024da.slice/crio-9282da63017c45fbefb01d3f33391be60226cd7624e78512e53e918d17727a3d WatchSource:0}: Error finding container 9282da63017c45fbefb01d3f33391be60226cd7624e78512e53e918d17727a3d: Status 404 returned error can't find the container with id 9282da63017c45fbefb01d3f33391be60226cd7624e78512e53e918d17727a3d Nov 03 00:14:06.768888 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:06.768852 5114 generic.go:356] "Generic (PLEG): container finished" podID="d2049967-6eed-4548-b61a-ea43e34024da" containerID="4421e9acbc91c4be78692eb864855d7662d4e2377a78488200b6b62f07659652" exitCode=0 Nov 03 00:14:06.769311 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:06.768933 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41df62c1-9ssfb" event={"ID":"d2049967-6eed-4548-b61a-ea43e34024da","Type":"ContainerDied","Data":"4421e9acbc91c4be78692eb864855d7662d4e2377a78488200b6b62f07659652"} Nov 03 00:14:06.769311 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:06.769003 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41df62c1-9ssfb" event={"ID":"d2049967-6eed-4548-b61a-ea43e34024da","Type":"ContainerStarted","Data":"9282da63017c45fbefb01d3f33391be60226cd7624e78512e53e918d17727a3d"} Nov 03 00:14:07.585192 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:14:07.585155 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:14:07.909677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:07.909647 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41df62c1-9ssfb" Nov 03 00:14:07.971979 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:07.971936 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2049967-6eed-4548-b61a-ea43e34024da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2049967-6eed-4548-b61a-ea43e34024da-maas-api-openshift-ca\") pod \"d2049967-6eed-4548-b61a-ea43e34024da\" (UID: \"d2049967-6eed-4548-b61a-ea43e34024da\") " Nov 03 00:14:07.972214 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:07.972001 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2049967-6eed-4548-b61a-ea43e34024da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2049967-6eed-4548-b61a-ea43e34024da-maas-api-xks-ca\") pod \"d2049967-6eed-4548-b61a-ea43e34024da\" (UID: \"d2049967-6eed-4548-b61a-ea43e34024da\") " Nov 03 00:14:07.972214 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:07.972058 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2049967-6eed-4548-b61a-ea43e34024da-kube-api-access-j8798\" (UniqueName: \"kubernetes.io/projected/d2049967-6eed-4548-b61a-ea43e34024da-kube-api-access-j8798\") pod \"d2049967-6eed-4548-b61a-ea43e34024da\" (UID: \"d2049967-6eed-4548-b61a-ea43e34024da\") " Nov 03 00:14:07.972403 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:07.972370 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2049967-6eed-4548-b61a-ea43e34024da-maas-api-openshift-ca" pod "d2049967-6eed-4548-b61a-ea43e34024da" (UID: "d2049967-6eed-4548-b61a-ea43e34024da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:14:07.974535 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:07.974455 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2049967-6eed-4548-b61a-ea43e34024da-maas-api-xks-ca" pod "d2049967-6eed-4548-b61a-ea43e34024da" (UID: "d2049967-6eed-4548-b61a-ea43e34024da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:14:07.974535 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:07.974516 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2049967-6eed-4548-b61a-ea43e34024da-kube-api-access-j8798" pod "d2049967-6eed-4548-b61a-ea43e34024da" (UID: "d2049967-6eed-4548-b61a-ea43e34024da"). InnerVolumeSpecName "kube-api-access-j8798". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:08.073784 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:08.073744 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2049967-6eed-4548-b61a-ea43e34024da-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:08.073784 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:08.073786 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2049967-6eed-4548-b61a-ea43e34024da-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:08.074010 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:08.073802 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8798\" (UniqueName: \"kubernetes.io/projected/d2049967-6eed-4548-b61a-ea43e34024da-kube-api-access-j8798\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:08.780714 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:08.780683 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41df62c1-9ssfb" Nov 03 00:14:08.780714 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:08.780692 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41df62c1-9ssfb" event={"ID":"d2049967-6eed-4548-b61a-ea43e34024da","Type":"ContainerDied","Data":"9282da63017c45fbefb01d3f33391be60226cd7624e78512e53e918d17727a3d"} Nov 03 00:14:08.780940 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:08.780736 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9282da63017c45fbefb01d3f33391be60226cd7624e78512e53e918d17727a3d" Nov 03 00:14:09.390861 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.390792 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:14:09.488219 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.488178 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-credential-socket\") pod \"14d58c32-6605-4191-a4cd-3a311461a25a\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " Nov 03 00:14:09.488457 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.488234 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-workload-socket\") pod \"14d58c32-6605-4191-a4cd-3a311461a25a\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " Nov 03 00:14:09.488457 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.488260 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14d58c32-6605-4191-a4cd-3a311461a25a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14d58c32-6605-4191-a4cd-3a311461a25a-istiod-ca-cert\") pod \"14d58c32-6605-4191-a4cd-3a311461a25a\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " Nov 03 00:14:09.488457 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.488290 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d58c32-6605-4191-a4cd-3a311461a25a-kube-api-access-xpxbg\" (UniqueName: \"kubernetes.io/projected/14d58c32-6605-4191-a4cd-3a311461a25a-kube-api-access-xpxbg\") pod \"14d58c32-6605-4191-a4cd-3a311461a25a\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " Nov 03 00:14:09.488457 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.488317 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-workload-certs\") pod \"14d58c32-6605-4191-a4cd-3a311461a25a\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " Nov 03 00:14:09.488457 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.488360 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-istio-envoy\") pod \"14d58c32-6605-4191-a4cd-3a311461a25a\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " Nov 03 00:14:09.488457 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.488389 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d58c32-6605-4191-a4cd-3a311461a25a-istio-token\" (UniqueName: \"kubernetes.io/projected/14d58c32-6605-4191-a4cd-3a311461a25a-istio-token\") pod \"14d58c32-6605-4191-a4cd-3a311461a25a\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " Nov 03 00:14:09.488877 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.488518 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14d58c32-6605-4191-a4cd-3a311461a25a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14d58c32-6605-4191-a4cd-3a311461a25a-istio-podinfo\") pod \"14d58c32-6605-4191-a4cd-3a311461a25a\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " Nov 03 00:14:09.488877 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.488551 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-workload-socket" pod "14d58c32-6605-4191-a4cd-3a311461a25a" (UID: "14d58c32-6605-4191-a4cd-3a311461a25a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:09.488877 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.488567 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-istio-data\") pod \"14d58c32-6605-4191-a4cd-3a311461a25a\" (UID: \"14d58c32-6605-4191-a4cd-3a311461a25a\") " Nov 03 00:14:09.488877 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.488595 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-credential-socket" pod "14d58c32-6605-4191-a4cd-3a311461a25a" (UID: "14d58c32-6605-4191-a4cd-3a311461a25a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:09.488877 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.488801 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-istio-data" pod "14d58c32-6605-4191-a4cd-3a311461a25a" (UID: "14d58c32-6605-4191-a4cd-3a311461a25a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:09.488877 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.488815 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d58c32-6605-4191-a4cd-3a311461a25a-istiod-ca-cert" pod "14d58c32-6605-4191-a4cd-3a311461a25a" (UID: "14d58c32-6605-4191-a4cd-3a311461a25a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:14:09.489171 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.489006 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-workload-certs" pod "14d58c32-6605-4191-a4cd-3a311461a25a" (UID: "14d58c32-6605-4191-a4cd-3a311461a25a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:09.489171 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.489149 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:09.489249 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.489173 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:09.489249 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.489189 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:09.489249 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.489202 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14d58c32-6605-4191-a4cd-3a311461a25a-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:09.489249 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.489218 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:09.490987 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.490958 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14d58c32-6605-4191-a4cd-3a311461a25a-istio-podinfo" pod "14d58c32-6605-4191-a4cd-3a311461a25a" (UID: "14d58c32-6605-4191-a4cd-3a311461a25a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:14:09.490987 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.490960 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d58c32-6605-4191-a4cd-3a311461a25a-istio-token" pod "14d58c32-6605-4191-a4cd-3a311461a25a" (UID: "14d58c32-6605-4191-a4cd-3a311461a25a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:09.491243 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.491215 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-istio-envoy" pod "14d58c32-6605-4191-a4cd-3a311461a25a" (UID: "14d58c32-6605-4191-a4cd-3a311461a25a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:09.491243 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.491229 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d58c32-6605-4191-a4cd-3a311461a25a-kube-api-access-xpxbg" pod "14d58c32-6605-4191-a4cd-3a311461a25a" (UID: "14d58c32-6605-4191-a4cd-3a311461a25a"). InnerVolumeSpecName "kube-api-access-xpxbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:09.590783 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.590747 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14d58c32-6605-4191-a4cd-3a311461a25a-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:09.590783 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.590778 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpxbg\" (UniqueName: \"kubernetes.io/projected/14d58c32-6605-4191-a4cd-3a311461a25a-kube-api-access-xpxbg\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:09.590783 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.590789 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14d58c32-6605-4191-a4cd-3a311461a25a-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:09.591051 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.590798 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14d58c32-6605-4191-a4cd-3a311461a25a-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:09.611347 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.611299 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh"] Nov 03 00:14:09.613803 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.613774 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr"] Nov 03 00:14:09.615962 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.615932 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx"] Nov 03 00:14:09.787449 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.787416 5114 generic.go:356] "Generic (PLEG): container finished" podID="14d58c32-6605-4191-a4cd-3a311461a25a" containerID="b11e70f7184e27f11fb038748dd53f54b6e19253b2b5e34b976f8a06b8df3c0a" exitCode=0 Nov 03 00:14:09.787755 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.787454 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" Nov 03 00:14:09.787755 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.787509 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" event={"ID":"14d58c32-6605-4191-a4cd-3a311461a25a","Type":"ContainerDied","Data":"b11e70f7184e27f11fb038748dd53f54b6e19253b2b5e34b976f8a06b8df3c0a"} Nov 03 00:14:09.787755 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.787555 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v" event={"ID":"14d58c32-6605-4191-a4cd-3a311461a25a","Type":"ContainerDied","Data":"2f8bfc6766c3d810a4cf2dfb94147421ea4b761406394c0f552d124b2b78c52e"} Nov 03 00:14:09.787755 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.787578 5114 scope.go:122] "RemoveContainer" containerID="b11e70f7184e27f11fb038748dd53f54b6e19253b2b5e34b976f8a06b8df3c0a" Nov 03 00:14:09.797695 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.797673 5114 scope.go:122] "RemoveContainer" containerID="b11e70f7184e27f11fb038748dd53f54b6e19253b2b5e34b976f8a06b8df3c0a" Nov 03 00:14:09.798056 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:14:09.798037 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b11e70f7184e27f11fb038748dd53f54b6e19253b2b5e34b976f8a06b8df3c0a\": container with ID starting with b11e70f7184e27f11fb038748dd53f54b6e19253b2b5e34b976f8a06b8df3c0a not found: ID does not exist" containerID="b11e70f7184e27f11fb038748dd53f54b6e19253b2b5e34b976f8a06b8df3c0a" Nov 03 00:14:09.798132 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:09.798067 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b11e70f7184e27f11fb038748dd53f54b6e19253b2b5e34b976f8a06b8df3c0a"} err="failed to get container status \"b11e70f7184e27f11fb038748dd53f54b6e19253b2b5e34b976f8a06b8df3c0a\": rpc error: code = NotFound desc = could not find container \"b11e70f7184e27f11fb038748dd53f54b6e19253b2b5e34b976f8a06b8df3c0a\": container with ID starting with b11e70f7184e27f11fb038748dd53f54b6e19253b2b5e34b976f8a06b8df3c0a not found: ID does not exist" Nov 03 00:14:10.069747 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:10.069659 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v"] Nov 03 00:14:10.073489 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:10.073450 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-65f4v"] Nov 03 00:14:10.792761 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:10.792723 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14d58c32-6605-4191-a4cd-3a311461a25a" path="/var/lib/kubelet/pods/14d58c32-6605-4191-a4cd-3a311461a25a/volumes" Nov 03 00:14:17.120549 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:17.120510 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d5a7914-4rnj2"] Nov 03 00:14:17.121001 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:17.120853 5114 cpu_manager.go:414] "containerMap: removing container" podUID="d2049967-6eed-4548-b61a-ea43e34024da" containerName="revoke-keys" Nov 03 00:14:17.121001 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:17.120863 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2049967-6eed-4548-b61a-ea43e34024da" containerName="revoke-keys" Nov 03 00:14:17.121001 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:17.120889 5114 cpu_manager.go:414] "containerMap: removing container" podUID="14d58c32-6605-4191-a4cd-3a311461a25a" containerName="istio-proxy" Nov 03 00:14:17.121001 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:17.120894 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14d58c32-6605-4191-a4cd-3a311461a25a" containerName="istio-proxy" Nov 03 00:14:17.121001 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:17.120946 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="14d58c32-6605-4191-a4cd-3a311461a25a" containerName="istio-proxy" Nov 03 00:14:17.121001 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:17.120967 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2049967-6eed-4548-b61a-ea43e34024da" containerName="revoke-keys" Nov 03 00:14:17.125612 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:17.125589 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a7914-4rnj2" Nov 03 00:14:17.229094 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:17.229057 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:14:17.229293 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:17.229203 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 03 00:14:17.230977 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:17.230950 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d5a7914-4rnj2"] Nov 03 00:14:17.269660 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:17.269616 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a4fa4e-577f-4740-ad90-bc62fd489044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d5a7914-4rnj2\" (UID: \"34a4fa4e-577f-4740-ad90-bc62fd489044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a7914-4rnj2" Nov 03 00:14:17.269660 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:17.269664 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a4fa4e-577f-4740-ad90-bc62fd489044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d5a7914-4rnj2\" (UID: \"34a4fa4e-577f-4740-ad90-bc62fd489044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a7914-4rnj2" Nov 03 00:14:17.269974 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:17.269691 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpb48\" (UniqueName: \"kubernetes.io/projected/34a4fa4e-577f-4740-ad90-bc62fd489044-kube-api-access-cpb48\") pod \"maas-api-revoke-sub-2d5a7914-4rnj2\" (UID: \"34a4fa4e-577f-4740-ad90-bc62fd489044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a7914-4rnj2" Nov 03 00:14:17.370346 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:17.370298 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a4fa4e-577f-4740-ad90-bc62fd489044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d5a7914-4rnj2\" (UID: \"34a4fa4e-577f-4740-ad90-bc62fd489044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a7914-4rnj2" Nov 03 00:14:17.370346 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:17.370349 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a4fa4e-577f-4740-ad90-bc62fd489044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d5a7914-4rnj2\" (UID: \"34a4fa4e-577f-4740-ad90-bc62fd489044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a7914-4rnj2" Nov 03 00:14:17.370715 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:17.370374 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpb48\" (UniqueName: \"kubernetes.io/projected/34a4fa4e-577f-4740-ad90-bc62fd489044-kube-api-access-cpb48\") pod \"maas-api-revoke-sub-2d5a7914-4rnj2\" (UID: \"34a4fa4e-577f-4740-ad90-bc62fd489044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a7914-4rnj2" Nov 03 00:14:17.371048 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:17.371028 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a4fa4e-577f-4740-ad90-bc62fd489044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d5a7914-4rnj2\" (UID: \"34a4fa4e-577f-4740-ad90-bc62fd489044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a7914-4rnj2" Nov 03 00:14:17.372906 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:17.372881 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a4fa4e-577f-4740-ad90-bc62fd489044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d5a7914-4rnj2\" (UID: \"34a4fa4e-577f-4740-ad90-bc62fd489044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a7914-4rnj2" Nov 03 00:14:17.443253 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:17.443217 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpb48\" (UniqueName: \"kubernetes.io/projected/34a4fa4e-577f-4740-ad90-bc62fd489044-kube-api-access-cpb48\") pod \"maas-api-revoke-sub-2d5a7914-4rnj2\" (UID: \"34a4fa4e-577f-4740-ad90-bc62fd489044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a7914-4rnj2" Nov 03 00:14:17.739182 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:17.739085 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a7914-4rnj2" Nov 03 00:14:18.112615 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:18.112587 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d5a7914-4rnj2"] Nov 03 00:14:18.115146 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:14:18.115115 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a4fa4e_577f_4740_ad90_bc62fd489044.slice/crio-5c54ca27bd017f7728c8f5e2449f2bfa745b755f61ebee473de243598c2d3925 WatchSource:0}: Error finding container 5c54ca27bd017f7728c8f5e2449f2bfa745b755f61ebee473de243598c2d3925: Status 404 returned error can't find the container with id 5c54ca27bd017f7728c8f5e2449f2bfa745b755f61ebee473de243598c2d3925 Nov 03 00:14:18.835211 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:18.835171 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a7914-4rnj2" event={"ID":"34a4fa4e-577f-4740-ad90-bc62fd489044","Type":"ContainerStarted","Data":"43fe4b1903702ba7a5a488eeb5018ba33586c0810f8ef881a7a602d4937488b1"} Nov 03 00:14:18.835693 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:18.835229 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a7914-4rnj2" event={"ID":"34a4fa4e-577f-4740-ad90-bc62fd489044","Type":"ContainerStarted","Data":"5c54ca27bd017f7728c8f5e2449f2bfa745b755f61ebee473de243598c2d3925"} Nov 03 00:14:18.951152 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:18.951093 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a7914-4rnj2" podStartSLOduration=1.951077666 podStartE2EDuration="1.951077666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:14:18.949964111 +0000 UTC m=+5134.302997493" watchObservedRunningTime="2026-11-03 00:14:18.951077666 +0000 UTC m=+5134.304111040" Nov 03 00:14:19.841253 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:19.841213 5114 generic.go:356] "Generic (PLEG): container finished" podID="34a4fa4e-577f-4740-ad90-bc62fd489044" containerID="43fe4b1903702ba7a5a488eeb5018ba33586c0810f8ef881a7a602d4937488b1" exitCode=0 Nov 03 00:14:19.841736 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:19.841298 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a7914-4rnj2" event={"ID":"34a4fa4e-577f-4740-ad90-bc62fd489044","Type":"ContainerDied","Data":"43fe4b1903702ba7a5a488eeb5018ba33586c0810f8ef881a7a602d4937488b1"} Nov 03 00:14:20.982660 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:20.982628 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a7914-4rnj2" Nov 03 00:14:21.103887 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:21.103811 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34a4fa4e-577f-4740-ad90-bc62fd489044-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a4fa4e-577f-4740-ad90-bc62fd489044-maas-api-openshift-ca\") pod \"34a4fa4e-577f-4740-ad90-bc62fd489044\" (UID: \"34a4fa4e-577f-4740-ad90-bc62fd489044\") " Nov 03 00:14:21.103887 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:21.103871 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34a4fa4e-577f-4740-ad90-bc62fd489044-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a4fa4e-577f-4740-ad90-bc62fd489044-maas-api-xks-ca\") pod \"34a4fa4e-577f-4740-ad90-bc62fd489044\" (UID: \"34a4fa4e-577f-4740-ad90-bc62fd489044\") " Nov 03 00:14:21.104144 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:21.103959 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a4fa4e-577f-4740-ad90-bc62fd489044-kube-api-access-cpb48\" (UniqueName: \"kubernetes.io/projected/34a4fa4e-577f-4740-ad90-bc62fd489044-kube-api-access-cpb48\") pod \"34a4fa4e-577f-4740-ad90-bc62fd489044\" (UID: \"34a4fa4e-577f-4740-ad90-bc62fd489044\") " Nov 03 00:14:21.104306 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:21.104276 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a4fa4e-577f-4740-ad90-bc62fd489044-maas-api-openshift-ca" pod "34a4fa4e-577f-4740-ad90-bc62fd489044" (UID: "34a4fa4e-577f-4740-ad90-bc62fd489044"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:14:21.106617 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:21.106583 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a4fa4e-577f-4740-ad90-bc62fd489044-kube-api-access-cpb48" pod "34a4fa4e-577f-4740-ad90-bc62fd489044" (UID: "34a4fa4e-577f-4740-ad90-bc62fd489044"). InnerVolumeSpecName "kube-api-access-cpb48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:21.107113 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:21.107084 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34a4fa4e-577f-4740-ad90-bc62fd489044-maas-api-xks-ca" pod "34a4fa4e-577f-4740-ad90-bc62fd489044" (UID: "34a4fa4e-577f-4740-ad90-bc62fd489044"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:14:21.205075 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:21.205027 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a4fa4e-577f-4740-ad90-bc62fd489044-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:21.205075 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:21.205063 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a4fa4e-577f-4740-ad90-bc62fd489044-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:21.205075 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:21.205076 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpb48\" (UniqueName: \"kubernetes.io/projected/34a4fa4e-577f-4740-ad90-bc62fd489044-kube-api-access-cpb48\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:21.852656 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:21.852616 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a7914-4rnj2" event={"ID":"34a4fa4e-577f-4740-ad90-bc62fd489044","Type":"ContainerDied","Data":"5c54ca27bd017f7728c8f5e2449f2bfa745b755f61ebee473de243598c2d3925"} Nov 03 00:14:21.852854 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:21.852666 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c54ca27bd017f7728c8f5e2449f2bfa745b755f61ebee473de243598c2d3925" Nov 03 00:14:21.852854 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:21.852628 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a7914-4rnj2" Nov 03 00:14:24.634266 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:24.634161 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" podUID="0e5fe23d-d96a-4099-a9bc-7844e2da3eb1" containerName="main" containerID="cri-o://d54e5edf32783c9861e1ff0237cc25adc511c6f78fa62c56f86438a4383d7f66" gracePeriod=45 Nov 03 00:14:24.635897 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:24.635836 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" podUID="496c6815-1b56-4082-9ffd-9ea51a762a58" containerName="main" containerID="cri-o://7e6bcfa1b5cbae8308584bb89eacf2bb44f7ad371e9676be247515182ea4243a" gracePeriod=45 Nov 03 00:14:24.638252 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:24.638192 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" podUID="bcd6362b-2995-4ebc-972d-dfcee6e41357" containerName="main" containerID="cri-o://654794705cd2e4f9b0c5e8e9fff6a2f3e41755ed3bb9a16b69248aedef951f2f" gracePeriod=45 Nov 03 00:14:24.638639 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:24.638474 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" podUID="0e5fe23d-d96a-4099-a9bc-7844e2da3eb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": EOF" Nov 03 00:14:24.869371 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:24.869333 5114 generic.go:356] "Generic (PLEG): container finished" podID="bcd6362b-2995-4ebc-972d-dfcee6e41357" containerID="654794705cd2e4f9b0c5e8e9fff6a2f3e41755ed3bb9a16b69248aedef951f2f" exitCode=0 Nov 03 00:14:24.869621 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:24.869408 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" event={"ID":"bcd6362b-2995-4ebc-972d-dfcee6e41357","Type":"ContainerDied","Data":"654794705cd2e4f9b0c5e8e9fff6a2f3e41755ed3bb9a16b69248aedef951f2f"} Nov 03 00:14:24.871639 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:24.871603 5114 generic.go:356] "Generic (PLEG): container finished" podID="496c6815-1b56-4082-9ffd-9ea51a762a58" containerID="7e6bcfa1b5cbae8308584bb89eacf2bb44f7ad371e9676be247515182ea4243a" exitCode=0 Nov 03 00:14:24.871822 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:24.871678 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" event={"ID":"496c6815-1b56-4082-9ffd-9ea51a762a58","Type":"ContainerDied","Data":"7e6bcfa1b5cbae8308584bb89eacf2bb44f7ad371e9676be247515182ea4243a"} Nov 03 00:14:24.873747 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:24.873694 5114 generic.go:356] "Generic (PLEG): container finished" podID="0e5fe23d-d96a-4099-a9bc-7844e2da3eb1" containerID="d54e5edf32783c9861e1ff0237cc25adc511c6f78fa62c56f86438a4383d7f66" exitCode=0 Nov 03 00:14:24.873747 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:24.873740 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" event={"ID":"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1","Type":"ContainerDied","Data":"d54e5edf32783c9861e1ff0237cc25adc511c6f78fa62c56f86438a4383d7f66"} Nov 03 00:14:24.900068 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:24.900037 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 03 00:14:24.937949 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:24.937918 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 03 00:14:24.940766 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:24.940744 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 03 00:14:25.042680 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.042651 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/496c6815-1b56-4082-9ffd-9ea51a762a58-tls-certs\" (UniqueName: \"kubernetes.io/secret/496c6815-1b56-4082-9ffd-9ea51a762a58-tls-certs\") pod \"496c6815-1b56-4082-9ffd-9ea51a762a58\" (UID: \"496c6815-1b56-4082-9ffd-9ea51a762a58\") " Nov 03 00:14:25.042885 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.042715 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-model-cache\") pod \"496c6815-1b56-4082-9ffd-9ea51a762a58\" (UID: \"496c6815-1b56-4082-9ffd-9ea51a762a58\") " Nov 03 00:14:25.042885 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.042734 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcd6362b-2995-4ebc-972d-dfcee6e41357-tls-certs\" (UniqueName: \"kubernetes.io/secret/bcd6362b-2995-4ebc-972d-dfcee6e41357-tls-certs\") pod \"bcd6362b-2995-4ebc-972d-dfcee6e41357\" (UID: \"bcd6362b-2995-4ebc-972d-dfcee6e41357\") " Nov 03 00:14:25.042885 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.042757 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-model-cache\") pod \"bcd6362b-2995-4ebc-972d-dfcee6e41357\" (UID: \"bcd6362b-2995-4ebc-972d-dfcee6e41357\") " Nov 03 00:14:25.042885 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.042775 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-dshm\") pod \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\" (UID: \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\") " Nov 03 00:14:25.042885 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.042813 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/496c6815-1b56-4082-9ffd-9ea51a762a58-kube-api-access-mffbw\" (UniqueName: \"kubernetes.io/projected/496c6815-1b56-4082-9ffd-9ea51a762a58-kube-api-access-mffbw\") pod \"496c6815-1b56-4082-9ffd-9ea51a762a58\" (UID: \"496c6815-1b56-4082-9ffd-9ea51a762a58\") " Nov 03 00:14:25.042885 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.042839 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-dshm\") pod \"bcd6362b-2995-4ebc-972d-dfcee6e41357\" (UID: \"bcd6362b-2995-4ebc-972d-dfcee6e41357\") " Nov 03 00:14:25.042885 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.042864 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-tmp-dir\") pod \"bcd6362b-2995-4ebc-972d-dfcee6e41357\" (UID: \"bcd6362b-2995-4ebc-972d-dfcee6e41357\") " Nov 03 00:14:25.043276 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.042907 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-tls-certs\" (UniqueName: \"kubernetes.io/secret/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-tls-certs\") pod \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\" (UID: \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\") " Nov 03 00:14:25.043276 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.042944 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-home\" (UniqueName: \"kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-home\") pod \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\" (UID: \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\") " Nov 03 00:14:25.043276 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.042980 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcd6362b-2995-4ebc-972d-dfcee6e41357-kube-api-access-x79b8\" (UniqueName: \"kubernetes.io/projected/bcd6362b-2995-4ebc-972d-dfcee6e41357-kube-api-access-x79b8\") pod \"bcd6362b-2995-4ebc-972d-dfcee6e41357\" (UID: \"bcd6362b-2995-4ebc-972d-dfcee6e41357\") " Nov 03 00:14:25.043276 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.043026 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-model-cache\") pod \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\" (UID: \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\") " Nov 03 00:14:25.043276 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.043054 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-tmp-dir\") pod \"496c6815-1b56-4082-9ffd-9ea51a762a58\" (UID: \"496c6815-1b56-4082-9ffd-9ea51a762a58\") " Nov 03 00:14:25.043276 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.043074 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-model-cache" pod "bcd6362b-2995-4ebc-972d-dfcee6e41357" (UID: "bcd6362b-2995-4ebc-972d-dfcee6e41357"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:25.043276 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.043079 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-model-cache" pod "496c6815-1b56-4082-9ffd-9ea51a762a58" (UID: "496c6815-1b56-4082-9ffd-9ea51a762a58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:25.043276 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.043090 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-home\" (UniqueName: \"kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-home\") pod \"bcd6362b-2995-4ebc-972d-dfcee6e41357\" (UID: \"bcd6362b-2995-4ebc-972d-dfcee6e41357\") " Nov 03 00:14:25.043276 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.043194 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-dshm\" (UniqueName: \"kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-dshm\") pod \"496c6815-1b56-4082-9ffd-9ea51a762a58\" (UID: \"496c6815-1b56-4082-9ffd-9ea51a762a58\") " Nov 03 00:14:25.043276 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.043276 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-tmp-dir\") pod \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\" (UID: \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\") " Nov 03 00:14:25.043834 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.043285 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-home" pod "bcd6362b-2995-4ebc-972d-dfcee6e41357" (UID: "bcd6362b-2995-4ebc-972d-dfcee6e41357"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:25.043834 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.043336 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-home\" (UniqueName: \"kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-home\") pod \"496c6815-1b56-4082-9ffd-9ea51a762a58\" (UID: \"496c6815-1b56-4082-9ffd-9ea51a762a58\") " Nov 03 00:14:25.043834 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.043360 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-model-cache" pod "0e5fe23d-d96a-4099-a9bc-7844e2da3eb1" (UID: "0e5fe23d-d96a-4099-a9bc-7844e2da3eb1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:25.043834 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.043403 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-kube-api-access-6h5mx\" (UniqueName: \"kubernetes.io/projected/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-kube-api-access-6h5mx\") pod \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\" (UID: \"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1\") " Nov 03 00:14:25.043834 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.043631 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-tmp-dir" pod "bcd6362b-2995-4ebc-972d-dfcee6e41357" (UID: "bcd6362b-2995-4ebc-972d-dfcee6e41357"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:25.043834 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.043652 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-tmp-dir" pod "496c6815-1b56-4082-9ffd-9ea51a762a58" (UID: "496c6815-1b56-4082-9ffd-9ea51a762a58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:25.043834 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.043804 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:25.043834 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.043822 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:25.043834 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.043837 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:25.044570 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.043853 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:25.044570 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.043868 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:25.044570 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.043882 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:25.044570 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.044098 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-tmp-dir" pod "0e5fe23d-d96a-4099-a9bc-7844e2da3eb1" (UID: "0e5fe23d-d96a-4099-a9bc-7844e2da3eb1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:25.044809 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.044645 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-home" pod "496c6815-1b56-4082-9ffd-9ea51a762a58" (UID: "496c6815-1b56-4082-9ffd-9ea51a762a58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:25.044809 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.044771 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-home" pod "0e5fe23d-d96a-4099-a9bc-7844e2da3eb1" (UID: "0e5fe23d-d96a-4099-a9bc-7844e2da3eb1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:25.045798 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.045767 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/496c6815-1b56-4082-9ffd-9ea51a762a58-tls-certs" pod "496c6815-1b56-4082-9ffd-9ea51a762a58" (UID: "496c6815-1b56-4082-9ffd-9ea51a762a58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:14:25.045931 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.045873 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcd6362b-2995-4ebc-972d-dfcee6e41357-tls-certs" pod "bcd6362b-2995-4ebc-972d-dfcee6e41357" (UID: "bcd6362b-2995-4ebc-972d-dfcee6e41357"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:14:25.047721 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.047691 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-dshm" pod "0e5fe23d-d96a-4099-a9bc-7844e2da3eb1" (UID: "0e5fe23d-d96a-4099-a9bc-7844e2da3eb1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:25.047721 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.047713 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-dshm" pod "496c6815-1b56-4082-9ffd-9ea51a762a58" (UID: "496c6815-1b56-4082-9ffd-9ea51a762a58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:25.047856 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.047719 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd6362b-2995-4ebc-972d-dfcee6e41357-kube-api-access-x79b8" pod "bcd6362b-2995-4ebc-972d-dfcee6e41357" (UID: "bcd6362b-2995-4ebc-972d-dfcee6e41357"). InnerVolumeSpecName "kube-api-access-x79b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:25.047856 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.047735 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-kube-api-access-6h5mx" pod "0e5fe23d-d96a-4099-a9bc-7844e2da3eb1" (UID: "0e5fe23d-d96a-4099-a9bc-7844e2da3eb1"). InnerVolumeSpecName "kube-api-access-6h5mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:25.047856 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.047720 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-dshm" pod "bcd6362b-2995-4ebc-972d-dfcee6e41357" (UID: "bcd6362b-2995-4ebc-972d-dfcee6e41357"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:25.047856 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.047802 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496c6815-1b56-4082-9ffd-9ea51a762a58-kube-api-access-mffbw" pod "496c6815-1b56-4082-9ffd-9ea51a762a58" (UID: "496c6815-1b56-4082-9ffd-9ea51a762a58"). InnerVolumeSpecName "kube-api-access-mffbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:25.048255 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.048241 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-tls-certs" pod "0e5fe23d-d96a-4099-a9bc-7844e2da3eb1" (UID: "0e5fe23d-d96a-4099-a9bc-7844e2da3eb1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:14:25.144975 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.144861 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcd6362b-2995-4ebc-972d-dfcee6e41357-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:25.144975 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.144901 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:25.144975 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.144915 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mffbw\" (UniqueName: \"kubernetes.io/projected/496c6815-1b56-4082-9ffd-9ea51a762a58-kube-api-access-mffbw\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:25.144975 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.144930 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcd6362b-2995-4ebc-972d-dfcee6e41357-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:25.144975 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.144941 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:25.144975 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.144952 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:25.144975 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.144974 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x79b8\" (UniqueName: \"kubernetes.io/projected/bcd6362b-2995-4ebc-972d-dfcee6e41357-kube-api-access-x79b8\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:25.145400 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.144987 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:25.145400 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.145001 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:25.145400 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.145013 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/496c6815-1b56-4082-9ffd-9ea51a762a58-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:25.145400 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.145026 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h5mx\" (UniqueName: \"kubernetes.io/projected/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1-kube-api-access-6h5mx\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:25.145400 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.145038 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/496c6815-1b56-4082-9ffd-9ea51a762a58-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:25.882321 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.882285 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" Nov 03 00:14:25.882321 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.882311 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx" event={"ID":"496c6815-1b56-4082-9ffd-9ea51a762a58","Type":"ContainerDied","Data":"11b4dfb560228d52c743412d02fedfe2e88fa187bf2adab1aa598c18f3c6c0ee"} Nov 03 00:14:25.882939 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.882375 5114 scope.go:122] "RemoveContainer" containerID="7e6bcfa1b5cbae8308584bb89eacf2bb44f7ad371e9676be247515182ea4243a" Nov 03 00:14:25.885525 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.884023 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" Nov 03 00:14:25.885525 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.884086 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh" event={"ID":"0e5fe23d-d96a-4099-a9bc-7844e2da3eb1","Type":"ContainerDied","Data":"7a83e18ed65c7ff6b1ca4fae6daae381a61431a46de119d8a022685056a19773"} Nov 03 00:14:25.893865 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.891684 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" event={"ID":"bcd6362b-2995-4ebc-972d-dfcee6e41357","Type":"ContainerDied","Data":"3d60a6f1e0acda76cf9b0da163834b144b21c0d6c44c3751d335189de1ab5d17"} Nov 03 00:14:25.893865 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.891725 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr" Nov 03 00:14:25.895473 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.895447 5114 scope.go:122] "RemoveContainer" containerID="d54e5edf32783c9861e1ff0237cc25adc511c6f78fa62c56f86438a4383d7f66" Nov 03 00:14:25.907617 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:25.907577 5114 scope.go:122] "RemoveContainer" containerID="654794705cd2e4f9b0c5e8e9fff6a2f3e41755ed3bb9a16b69248aedef951f2f" Nov 03 00:14:26.046818 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:26.046777 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh"] Nov 03 00:14:26.096204 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:26.096170 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/distinct-2-w2-08c2c6-kserve-fd75bd8b4-phmxh"] Nov 03 00:14:26.206002 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:26.205906 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr"] Nov 03 00:14:26.209589 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:26.209547 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/premium-w2-08c2c6-kserve-57db6b8f5b-wh9lr"] Nov 03 00:14:26.317600 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:26.317553 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx"] Nov 03 00:14:26.322981 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:26.322937 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/simulator-w2-08c2c6-kserve-6b7769c645-bb5xx"] Nov 03 00:14:26.791869 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:26.791835 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e5fe23d-d96a-4099-a9bc-7844e2da3eb1" path="/var/lib/kubelet/pods/0e5fe23d-d96a-4099-a9bc-7844e2da3eb1/volumes" Nov 03 00:14:26.792243 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:26.792228 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="496c6815-1b56-4082-9ffd-9ea51a762a58" path="/var/lib/kubelet/pods/496c6815-1b56-4082-9ffd-9ea51a762a58/volumes" Nov 03 00:14:26.792604 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:26.792590 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcd6362b-2995-4ebc-972d-dfcee6e41357" path="/var/lib/kubelet/pods/bcd6362b-2995-4ebc-972d-dfcee6e41357/volumes" Nov 03 00:14:36.045752 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:36.045709 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d20fd8ec-dgrx2"] Nov 03 00:14:36.095276 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:36.095237 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d20fd8ec-dgrx2"] Nov 03 00:14:36.792040 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:36.792003 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="874f6db0-2ff4-44b1-a194-c89239749458" path="/var/lib/kubelet/pods/874f6db0-2ff4-44b1-a194-c89239749458/volumes" Nov 03 00:14:37.584886 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:14:37.584826 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:14:48.046177 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:48.046125 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-283112ef-cr828"] Nov 03 00:14:48.096341 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:48.096309 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-283112ef-cr828"] Nov 03 00:14:48.792420 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:48.792380 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4ef303d-dc7d-41e7-a795-269968a5f7a2" path="/var/lib/kubelet/pods/c4ef303d-dc7d-41e7-a795-269968a5f7a2/volumes" Nov 03 00:14:50.864498 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:50.864453 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7"] Nov 03 00:14:50.864942 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:50.864730 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" podUID="d2cc2d36-def2-45fb-86bf-b35f56a969e6" containerName="istio-proxy" containerID="cri-o://589c21f68ad0ee71280b448cf87e25e326a3bda5d18b78f7cd3b5e658662fc6b" gracePeriod=30 Nov 03 00:14:56.045755 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.045719 5114 generic.go:356] "Generic (PLEG): container finished" podID="d2cc2d36-def2-45fb-86bf-b35f56a969e6" containerID="589c21f68ad0ee71280b448cf87e25e326a3bda5d18b78f7cd3b5e658662fc6b" exitCode=0 Nov 03 00:14:56.046209 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.045794 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" event={"ID":"d2cc2d36-def2-45fb-86bf-b35f56a969e6","Type":"ContainerDied","Data":"589c21f68ad0ee71280b448cf87e25e326a3bda5d18b78f7cd3b5e658662fc6b"} Nov 03 00:14:56.046209 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.045862 5114 scope.go:122] "RemoveContainer" containerID="a761c07dd672a89d09871c4ad763be5755bed6ad2a655a8a8b300f3a6323eb8b" Nov 03 00:14:56.111506 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.111450 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 03 00:14:56.193141 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.193043 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2cc2d36-def2-45fb-86bf-b35f56a969e6-kube-api-access-qzn4t\" (UniqueName: \"kubernetes.io/projected/d2cc2d36-def2-45fb-86bf-b35f56a969e6-kube-api-access-qzn4t\") pod \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " Nov 03 00:14:56.193141 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.193102 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-credential-socket\") pod \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " Nov 03 00:14:56.193403 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.193149 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istiod-ca-cert\") pod \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " Nov 03 00:14:56.193403 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.193187 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-workload-socket\") pod \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " Nov 03 00:14:56.193403 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.193227 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-podinfo\") pod \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " Nov 03 00:14:56.193403 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.193297 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-data\") pod \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " Nov 03 00:14:56.193403 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.193328 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-workload-certs\") pod \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " Nov 03 00:14:56.193403 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.193355 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-envoy\") pod \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " Nov 03 00:14:56.193403 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.193380 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-token\" (UniqueName: \"kubernetes.io/projected/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-token\") pod \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\" (UID: \"d2cc2d36-def2-45fb-86bf-b35f56a969e6\") " Nov 03 00:14:56.193854 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.193546 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-credential-socket" pod "d2cc2d36-def2-45fb-86bf-b35f56a969e6" (UID: "d2cc2d36-def2-45fb-86bf-b35f56a969e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:56.193854 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.193566 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-workload-socket" pod "d2cc2d36-def2-45fb-86bf-b35f56a969e6" (UID: "d2cc2d36-def2-45fb-86bf-b35f56a969e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:56.193854 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.193661 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istiod-ca-cert" pod "d2cc2d36-def2-45fb-86bf-b35f56a969e6" (UID: "d2cc2d36-def2-45fb-86bf-b35f56a969e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:14:56.193854 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.193792 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:56.193854 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.193808 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:56.193854 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.193817 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:56.193854 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.193825 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-data" pod "d2cc2d36-def2-45fb-86bf-b35f56a969e6" (UID: "d2cc2d36-def2-45fb-86bf-b35f56a969e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:56.194147 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.193895 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-workload-certs" pod "d2cc2d36-def2-45fb-86bf-b35f56a969e6" (UID: "d2cc2d36-def2-45fb-86bf-b35f56a969e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:56.195711 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.195681 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-envoy" pod "d2cc2d36-def2-45fb-86bf-b35f56a969e6" (UID: "d2cc2d36-def2-45fb-86bf-b35f56a969e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:56.195895 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.195847 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2cc2d36-def2-45fb-86bf-b35f56a969e6-kube-api-access-qzn4t" pod "d2cc2d36-def2-45fb-86bf-b35f56a969e6" (UID: "d2cc2d36-def2-45fb-86bf-b35f56a969e6"). InnerVolumeSpecName "kube-api-access-qzn4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:56.196147 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.196118 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-podinfo" pod "d2cc2d36-def2-45fb-86bf-b35f56a969e6" (UID: "d2cc2d36-def2-45fb-86bf-b35f56a969e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:14:56.196147 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.196132 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-token" pod "d2cc2d36-def2-45fb-86bf-b35f56a969e6" (UID: "d2cc2d36-def2-45fb-86bf-b35f56a969e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:56.294703 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.294663 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:56.294703 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.294696 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:56.294703 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.294707 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:56.294961 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.294717 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:56.294961 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.294727 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzn4t\" (UniqueName: \"kubernetes.io/projected/d2cc2d36-def2-45fb-86bf-b35f56a969e6-kube-api-access-qzn4t\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:56.294961 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:56.294736 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2cc2d36-def2-45fb-86bf-b35f56a969e6-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:57.052021 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:57.051936 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" Nov 03 00:14:57.052021 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:57.051968 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7" event={"ID":"d2cc2d36-def2-45fb-86bf-b35f56a969e6","Type":"ContainerDied","Data":"bcdfd16844dc8de03beca527fe0dd26b00ec915a0ee11d0aaf7c6b175cab1225"} Nov 03 00:14:57.052618 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:57.052032 5114 scope.go:122] "RemoveContainer" containerID="589c21f68ad0ee71280b448cf87e25e326a3bda5d18b78f7cd3b5e658662fc6b" Nov 03 00:14:57.264451 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:57.264409 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7"] Nov 03 00:14:57.314164 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:57.314072 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-86dr7"] Nov 03 00:14:58.792363 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:14:58.792326 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2cc2d36-def2-45fb-86bf-b35f56a969e6" path="/var/lib/kubelet/pods/d2cc2d36-def2-45fb-86bf-b35f56a969e6/volumes" Nov 03 00:15:00.167145 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.167104 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz"] Nov 03 00:15:00.167719 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.167655 5114 cpu_manager.go:414] "containerMap: removing container" podUID="d2cc2d36-def2-45fb-86bf-b35f56a969e6" containerName="istio-proxy" Nov 03 00:15:00.167719 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.167676 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2cc2d36-def2-45fb-86bf-b35f56a969e6" containerName="istio-proxy" Nov 03 00:15:00.167719 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.167689 5114 cpu_manager.go:414] "containerMap: removing container" podUID="496c6815-1b56-4082-9ffd-9ea51a762a58" containerName="main" Nov 03 00:15:00.167719 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.167695 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="496c6815-1b56-4082-9ffd-9ea51a762a58" containerName="main" Nov 03 00:15:00.167719 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.167712 5114 cpu_manager.go:414] "containerMap: removing container" podUID="34a4fa4e-577f-4740-ad90-bc62fd489044" containerName="revoke-keys" Nov 03 00:15:00.168010 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.167720 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a4fa4e-577f-4740-ad90-bc62fd489044" containerName="revoke-keys" Nov 03 00:15:00.168010 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.167752 5114 cpu_manager.go:414] "containerMap: removing container" podUID="bcd6362b-2995-4ebc-972d-dfcee6e41357" containerName="main" Nov 03 00:15:00.168010 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.167760 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcd6362b-2995-4ebc-972d-dfcee6e41357" containerName="main" Nov 03 00:15:00.168010 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.167771 5114 cpu_manager.go:414] "containerMap: removing container" podUID="d2cc2d36-def2-45fb-86bf-b35f56a969e6" containerName="istio-proxy" Nov 03 00:15:00.168010 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.167778 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2cc2d36-def2-45fb-86bf-b35f56a969e6" containerName="istio-proxy" Nov 03 00:15:00.168010 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.167792 5114 cpu_manager.go:414] "containerMap: removing container" podUID="0e5fe23d-d96a-4099-a9bc-7844e2da3eb1" containerName="main" Nov 03 00:15:00.168010 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.167799 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e5fe23d-d96a-4099-a9bc-7844e2da3eb1" containerName="main" Nov 03 00:15:00.168010 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.167883 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="34a4fa4e-577f-4740-ad90-bc62fd489044" containerName="revoke-keys" Nov 03 00:15:00.168010 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.167899 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcd6362b-2995-4ebc-972d-dfcee6e41357" containerName="main" Nov 03 00:15:00.168010 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.167907 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2cc2d36-def2-45fb-86bf-b35f56a969e6" containerName="istio-proxy" Nov 03 00:15:00.168010 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.167915 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e5fe23d-d96a-4099-a9bc-7844e2da3eb1" containerName="main" Nov 03 00:15:00.168010 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.167926 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2cc2d36-def2-45fb-86bf-b35f56a969e6" containerName="istio-proxy" Nov 03 00:15:00.168010 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.167935 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="496c6815-1b56-4082-9ffd-9ea51a762a58" containerName="main" Nov 03 00:15:00.171286 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.171262 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq"] Nov 03 00:15:00.171451 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.171380 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz" Nov 03 00:15:00.174678 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.174655 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq" Nov 03 00:15:00.223347 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.223311 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 03 00:15:00.230719 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.230686 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b7pk\" (UniqueName: \"kubernetes.io/projected/7430ed00-5e37-4edb-96a5-d43ab703ea71-kube-api-access-6b7pk\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz\" (UID: \"7430ed00-5e37-4edb-96a5-d43ab703ea71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz" Nov 03 00:15:00.230849 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.230740 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nsvf\" (UniqueName: \"kubernetes.io/projected/3bf9431d-e50a-4d0c-83c4-015cbad7beeb-kube-api-access-6nsvf\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq\" (UID: \"3bf9431d-e50a-4d0c-83c4-015cbad7beeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq" Nov 03 00:15:00.230904 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.230885 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf9431d-e50a-4d0c-83c4-015cbad7beeb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq\" (UID: \"3bf9431d-e50a-4d0c-83c4-015cbad7beeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq" Nov 03 00:15:00.230944 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.230933 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7430ed00-5e37-4edb-96a5-d43ab703ea71-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz\" (UID: \"7430ed00-5e37-4edb-96a5-d43ab703ea71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz" Nov 03 00:15:00.275635 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.275551 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq"] Nov 03 00:15:00.277566 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.277535 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz"] Nov 03 00:15:00.332364 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.332322 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b7pk\" (UniqueName: \"kubernetes.io/projected/7430ed00-5e37-4edb-96a5-d43ab703ea71-kube-api-access-6b7pk\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz\" (UID: \"7430ed00-5e37-4edb-96a5-d43ab703ea71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz" Nov 03 00:15:00.332364 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.332368 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nsvf\" (UniqueName: \"kubernetes.io/projected/3bf9431d-e50a-4d0c-83c4-015cbad7beeb-kube-api-access-6nsvf\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq\" (UID: \"3bf9431d-e50a-4d0c-83c4-015cbad7beeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq" Nov 03 00:15:00.332635 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.332424 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf9431d-e50a-4d0c-83c4-015cbad7beeb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq\" (UID: \"3bf9431d-e50a-4d0c-83c4-015cbad7beeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq" Nov 03 00:15:00.332635 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.332452 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7430ed00-5e37-4edb-96a5-d43ab703ea71-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz\" (UID: \"7430ed00-5e37-4edb-96a5-d43ab703ea71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz" Nov 03 00:15:00.333325 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.333302 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7430ed00-5e37-4edb-96a5-d43ab703ea71-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz\" (UID: \"7430ed00-5e37-4edb-96a5-d43ab703ea71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz" Nov 03 00:15:00.333370 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.333300 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf9431d-e50a-4d0c-83c4-015cbad7beeb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq\" (UID: \"3bf9431d-e50a-4d0c-83c4-015cbad7beeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq" Nov 03 00:15:00.393576 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.393541 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nsvf\" (UniqueName: \"kubernetes.io/projected/3bf9431d-e50a-4d0c-83c4-015cbad7beeb-kube-api-access-6nsvf\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq\" (UID: \"3bf9431d-e50a-4d0c-83c4-015cbad7beeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq" Nov 03 00:15:00.393745 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.393544 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b7pk\" (UniqueName: \"kubernetes.io/projected/7430ed00-5e37-4edb-96a5-d43ab703ea71-kube-api-access-6b7pk\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz\" (UID: \"7430ed00-5e37-4edb-96a5-d43ab703ea71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz" Nov 03 00:15:00.486064 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.485978 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz" Nov 03 00:15:00.491217 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.491195 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq" Nov 03 00:15:00.858970 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.858940 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz"] Nov 03 00:15:00.861299 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:15:00.861274 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7430ed00_5e37_4edb_96a5_d43ab703ea71.slice/crio-01898bd96de7c8d3ea3d86bc094e54197f9046e3bbdcbc5e41ced3b74051ae82 WatchSource:0}: Error finding container 01898bd96de7c8d3ea3d86bc094e54197f9046e3bbdcbc5e41ced3b74051ae82: Status 404 returned error can't find the container with id 01898bd96de7c8d3ea3d86bc094e54197f9046e3bbdcbc5e41ced3b74051ae82 Nov 03 00:15:00.910135 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:00.910094 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq"] Nov 03 00:15:00.912158 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:15:00.912130 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf9431d_e50a_4d0c_83c4_015cbad7beeb.slice/crio-2274aba3978e7bc52df3902187dfd141cdb9f131cd5484e0acf032ba66af0f97 WatchSource:0}: Error finding container 2274aba3978e7bc52df3902187dfd141cdb9f131cd5484e0acf032ba66af0f97: Status 404 returned error can't find the container with id 2274aba3978e7bc52df3902187dfd141cdb9f131cd5484e0acf032ba66af0f97 Nov 03 00:15:01.073496 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:01.073447 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz" event={"ID":"7430ed00-5e37-4edb-96a5-d43ab703ea71","Type":"ContainerStarted","Data":"01898bd96de7c8d3ea3d86bc094e54197f9046e3bbdcbc5e41ced3b74051ae82"} Nov 03 00:15:01.074661 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:01.074627 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq" event={"ID":"3bf9431d-e50a-4d0c-83c4-015cbad7beeb","Type":"ContainerStarted","Data":"2274aba3978e7bc52df3902187dfd141cdb9f131cd5484e0acf032ba66af0f97"} Nov 03 00:15:02.081543 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:02.081471 5114 generic.go:356] "Generic (PLEG): container finished" podID="7430ed00-5e37-4edb-96a5-d43ab703ea71" containerID="08c961afcbc7d06f7de8a99a6be3fd37557cb32f74bd5e830ddc1e9726fb5e4c" exitCode=60 Nov 03 00:15:02.082019 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:02.081562 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz" event={"ID":"7430ed00-5e37-4edb-96a5-d43ab703ea71","Type":"ContainerDied","Data":"08c961afcbc7d06f7de8a99a6be3fd37557cb32f74bd5e830ddc1e9726fb5e4c"} Nov 03 00:15:02.082019 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:02.081884 5114 scope.go:122] "RemoveContainer" containerID="08c961afcbc7d06f7de8a99a6be3fd37557cb32f74bd5e830ddc1e9726fb5e4c" Nov 03 00:15:02.084037 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:02.084009 5114 generic.go:356] "Generic (PLEG): container finished" podID="3bf9431d-e50a-4d0c-83c4-015cbad7beeb" containerID="6e8d9517023b510bddd86d0f0baf4e9b3324dcafa832a0d7bd834e24176ada97" exitCode=60 Nov 03 00:15:02.084154 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:02.084084 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq" event={"ID":"3bf9431d-e50a-4d0c-83c4-015cbad7beeb","Type":"ContainerDied","Data":"6e8d9517023b510bddd86d0f0baf4e9b3324dcafa832a0d7bd834e24176ada97"} Nov 03 00:15:02.084381 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:02.084364 5114 scope.go:122] "RemoveContainer" containerID="6e8d9517023b510bddd86d0f0baf4e9b3324dcafa832a0d7bd834e24176ada97" Nov 03 00:15:03.091335 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:03.091297 5114 generic.go:356] "Generic (PLEG): container finished" podID="3bf9431d-e50a-4d0c-83c4-015cbad7beeb" containerID="8c5d9e6ef7a4d8b1f515dbd11d5a2529454b36ec011289920539bc49a4f7a723" exitCode=60 Nov 03 00:15:03.091861 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:03.091387 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq" event={"ID":"3bf9431d-e50a-4d0c-83c4-015cbad7beeb","Type":"ContainerDied","Data":"8c5d9e6ef7a4d8b1f515dbd11d5a2529454b36ec011289920539bc49a4f7a723"} Nov 03 00:15:03.091861 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:03.091464 5114 scope.go:122] "RemoveContainer" containerID="6e8d9517023b510bddd86d0f0baf4e9b3324dcafa832a0d7bd834e24176ada97" Nov 03 00:15:03.091861 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:03.091768 5114 scope.go:122] "RemoveContainer" containerID="8c5d9e6ef7a4d8b1f515dbd11d5a2529454b36ec011289920539bc49a4f7a723" Nov 03 00:15:03.092085 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:15:03.092061 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq_odh-ai-gateway-infra(3bf9431d-e50a-4d0c-83c4-015cbad7beeb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq" podUID="3bf9431d-e50a-4d0c-83c4-015cbad7beeb" Nov 03 00:15:03.093453 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:03.093428 5114 generic.go:356] "Generic (PLEG): container finished" podID="7430ed00-5e37-4edb-96a5-d43ab703ea71" containerID="31778a86ebd385ae4433143faf7436eab9046b0ae1b07b1db12d7497d0d0048c" exitCode=60 Nov 03 00:15:03.093569 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:03.093511 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz" event={"ID":"7430ed00-5e37-4edb-96a5-d43ab703ea71","Type":"ContainerDied","Data":"31778a86ebd385ae4433143faf7436eab9046b0ae1b07b1db12d7497d0d0048c"} Nov 03 00:15:03.093750 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:03.093738 5114 scope.go:122] "RemoveContainer" containerID="31778a86ebd385ae4433143faf7436eab9046b0ae1b07b1db12d7497d0d0048c" Nov 03 00:15:03.093945 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:15:03.093930 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz_odh-ai-gateway-infra(7430ed00-5e37-4edb-96a5-d43ab703ea71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz" podUID="7430ed00-5e37-4edb-96a5-d43ab703ea71" Nov 03 00:15:03.102652 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:03.102615 5114 scope.go:122] "RemoveContainer" containerID="08c961afcbc7d06f7de8a99a6be3fd37557cb32f74bd5e830ddc1e9726fb5e4c" Nov 03 00:15:04.101116 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:04.101085 5114 scope.go:122] "RemoveContainer" containerID="8c5d9e6ef7a4d8b1f515dbd11d5a2529454b36ec011289920539bc49a4f7a723" Nov 03 00:15:04.101605 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:15:04.101354 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq_odh-ai-gateway-infra(3bf9431d-e50a-4d0c-83c4-015cbad7beeb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq" podUID="3bf9431d-e50a-4d0c-83c4-015cbad7beeb" Nov 03 00:15:04.102804 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:04.102783 5114 scope.go:122] "RemoveContainer" containerID="31778a86ebd385ae4433143faf7436eab9046b0ae1b07b1db12d7497d0d0048c" Nov 03 00:15:04.103027 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:15:04.103008 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz_odh-ai-gateway-infra(7430ed00-5e37-4edb-96a5-d43ab703ea71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz" podUID="7430ed00-5e37-4edb-96a5-d43ab703ea71" Nov 03 00:15:07.583379 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:15:07.583333 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:15:15.787594 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:15.787554 5114 scope.go:122] "RemoveContainer" containerID="31778a86ebd385ae4433143faf7436eab9046b0ae1b07b1db12d7497d0d0048c" Nov 03 00:15:15.788109 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:15.787659 5114 scope.go:122] "RemoveContainer" containerID="8c5d9e6ef7a4d8b1f515dbd11d5a2529454b36ec011289920539bc49a4f7a723" Nov 03 00:15:16.163549 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:16.163503 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq" event={"ID":"3bf9431d-e50a-4d0c-83c4-015cbad7beeb","Type":"ContainerStarted","Data":"cff04e07e2c35c5a7f34d996a812776dbcbfdd191b7c046924780c6e783d76ce"} Nov 03 00:15:16.165731 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:16.165698 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz" event={"ID":"7430ed00-5e37-4edb-96a5-d43ab703ea71","Type":"ContainerStarted","Data":"5a67e1a57f2c4ea35716b0c4415b25cd3d8d103e1798ae17383fc6ea06508199"} Nov 03 00:15:16.279163 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:16.279110 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq" podStartSLOduration=16.279095064 podStartE2EDuration="16.279095064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:16.27739367 +0000 UTC m=+5191.630427048" watchObservedRunningTime="2026-11-03 00:15:16.279095064 +0000 UTC m=+5191.632128439" Nov 03 00:15:16.392933 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:16.392823 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz" podStartSLOduration=16.39280839 podStartE2EDuration="16.39280839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:16.392096061 +0000 UTC m=+5191.745129438" watchObservedRunningTime="2026-11-03 00:15:16.39280839 +0000 UTC m=+5191.745841764" Nov 03 00:15:17.171950 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:17.171914 5114 generic.go:356] "Generic (PLEG): container finished" podID="3bf9431d-e50a-4d0c-83c4-015cbad7beeb" containerID="cff04e07e2c35c5a7f34d996a812776dbcbfdd191b7c046924780c6e783d76ce" exitCode=60 Nov 03 00:15:17.172465 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:17.171991 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq" event={"ID":"3bf9431d-e50a-4d0c-83c4-015cbad7beeb","Type":"ContainerDied","Data":"cff04e07e2c35c5a7f34d996a812776dbcbfdd191b7c046924780c6e783d76ce"} Nov 03 00:15:17.172465 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:17.172056 5114 scope.go:122] "RemoveContainer" containerID="8c5d9e6ef7a4d8b1f515dbd11d5a2529454b36ec011289920539bc49a4f7a723" Nov 03 00:15:17.172465 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:17.172418 5114 scope.go:122] "RemoveContainer" containerID="cff04e07e2c35c5a7f34d996a812776dbcbfdd191b7c046924780c6e783d76ce" Nov 03 00:15:17.172759 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:15:17.172734 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq_odh-ai-gateway-infra(3bf9431d-e50a-4d0c-83c4-015cbad7beeb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq" podUID="3bf9431d-e50a-4d0c-83c4-015cbad7beeb" Nov 03 00:15:17.173958 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:17.173936 5114 generic.go:356] "Generic (PLEG): container finished" podID="7430ed00-5e37-4edb-96a5-d43ab703ea71" containerID="5a67e1a57f2c4ea35716b0c4415b25cd3d8d103e1798ae17383fc6ea06508199" exitCode=60 Nov 03 00:15:17.174110 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:17.173999 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz" event={"ID":"7430ed00-5e37-4edb-96a5-d43ab703ea71","Type":"ContainerDied","Data":"5a67e1a57f2c4ea35716b0c4415b25cd3d8d103e1798ae17383fc6ea06508199"} Nov 03 00:15:17.174386 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:17.174370 5114 scope.go:122] "RemoveContainer" containerID="5a67e1a57f2c4ea35716b0c4415b25cd3d8d103e1798ae17383fc6ea06508199" Nov 03 00:15:17.174636 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:15:17.174618 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz_odh-ai-gateway-infra(7430ed00-5e37-4edb-96a5-d43ab703ea71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz" podUID="7430ed00-5e37-4edb-96a5-d43ab703ea71" Nov 03 00:15:17.184100 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:17.184071 5114 scope.go:122] "RemoveContainer" containerID="31778a86ebd385ae4433143faf7436eab9046b0ae1b07b1db12d7497d0d0048c" Nov 03 00:15:17.286684 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:17.286644 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq"] Nov 03 00:15:17.402142 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:17.402104 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz"] Nov 03 00:15:18.338064 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:18.338032 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq" Nov 03 00:15:18.364974 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:18.364933 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz" Nov 03 00:15:18.498886 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:18.498791 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7430ed00-5e37-4edb-96a5-d43ab703ea71-kube-api-access-6b7pk\" (UniqueName: \"kubernetes.io/projected/7430ed00-5e37-4edb-96a5-d43ab703ea71-kube-api-access-6b7pk\") pod \"7430ed00-5e37-4edb-96a5-d43ab703ea71\" (UID: \"7430ed00-5e37-4edb-96a5-d43ab703ea71\") " Nov 03 00:15:18.498886 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:18.498828 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7430ed00-5e37-4edb-96a5-d43ab703ea71-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7430ed00-5e37-4edb-96a5-d43ab703ea71-maas-api-service-ca\") pod \"7430ed00-5e37-4edb-96a5-d43ab703ea71\" (UID: \"7430ed00-5e37-4edb-96a5-d43ab703ea71\") " Nov 03 00:15:18.498886 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:18.498869 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf9431d-e50a-4d0c-83c4-015cbad7beeb-kube-api-access-6nsvf\" (UniqueName: \"kubernetes.io/projected/3bf9431d-e50a-4d0c-83c4-015cbad7beeb-kube-api-access-6nsvf\") pod \"3bf9431d-e50a-4d0c-83c4-015cbad7beeb\" (UID: \"3bf9431d-e50a-4d0c-83c4-015cbad7beeb\") " Nov 03 00:15:18.499154 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:18.498920 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf9431d-e50a-4d0c-83c4-015cbad7beeb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf9431d-e50a-4d0c-83c4-015cbad7beeb-maas-api-service-ca\") pod \"3bf9431d-e50a-4d0c-83c4-015cbad7beeb\" (UID: \"3bf9431d-e50a-4d0c-83c4-015cbad7beeb\") " Nov 03 00:15:18.499265 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:18.499234 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7430ed00-5e37-4edb-96a5-d43ab703ea71-maas-api-service-ca" pod "7430ed00-5e37-4edb-96a5-d43ab703ea71" (UID: "7430ed00-5e37-4edb-96a5-d43ab703ea71"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:18.499325 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:18.499294 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf9431d-e50a-4d0c-83c4-015cbad7beeb-maas-api-service-ca" pod "3bf9431d-e50a-4d0c-83c4-015cbad7beeb" (UID: "3bf9431d-e50a-4d0c-83c4-015cbad7beeb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:18.501187 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:18.501157 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7430ed00-5e37-4edb-96a5-d43ab703ea71-kube-api-access-6b7pk" pod "7430ed00-5e37-4edb-96a5-d43ab703ea71" (UID: "7430ed00-5e37-4edb-96a5-d43ab703ea71"). InnerVolumeSpecName "kube-api-access-6b7pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:18.501187 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:18.501172 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf9431d-e50a-4d0c-83c4-015cbad7beeb-kube-api-access-6nsvf" pod "3bf9431d-e50a-4d0c-83c4-015cbad7beeb" (UID: "3bf9431d-e50a-4d0c-83c4-015cbad7beeb"). InnerVolumeSpecName "kube-api-access-6nsvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:18.600930 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:18.600892 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf9431d-e50a-4d0c-83c4-015cbad7beeb-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:18.600930 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:18.600927 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b7pk\" (UniqueName: \"kubernetes.io/projected/7430ed00-5e37-4edb-96a5-d43ab703ea71-kube-api-access-6b7pk\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:18.600930 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:18.600939 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7430ed00-5e37-4edb-96a5-d43ab703ea71-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:18.601259 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:18.600950 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nsvf\" (UniqueName: \"kubernetes.io/projected/3bf9431d-e50a-4d0c-83c4-015cbad7beeb-kube-api-access-6nsvf\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:19.192886 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:19.192795 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq" Nov 03 00:15:19.192886 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:19.192842 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq" event={"ID":"3bf9431d-e50a-4d0c-83c4-015cbad7beeb","Type":"ContainerDied","Data":"2274aba3978e7bc52df3902187dfd141cdb9f131cd5484e0acf032ba66af0f97"} Nov 03 00:15:19.193167 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:19.192910 5114 scope.go:122] "RemoveContainer" containerID="cff04e07e2c35c5a7f34d996a812776dbcbfdd191b7c046924780c6e783d76ce" Nov 03 00:15:19.194379 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:19.194348 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz" Nov 03 00:15:19.194379 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:19.194353 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz" event={"ID":"7430ed00-5e37-4edb-96a5-d43ab703ea71","Type":"ContainerDied","Data":"01898bd96de7c8d3ea3d86bc094e54197f9046e3bbdcbc5e41ced3b74051ae82"} Nov 03 00:15:19.201874 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:19.201851 5114 scope.go:122] "RemoveContainer" containerID="5a67e1a57f2c4ea35716b0c4415b25cd3d8d103e1798ae17383fc6ea06508199" Nov 03 00:15:19.356318 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:19.356281 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq"] Nov 03 00:15:19.405775 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:19.405732 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3bee1b-29894415-pnhpq"] Nov 03 00:15:19.472702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:19.472617 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz"] Nov 03 00:15:19.476851 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:19.476803 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9b6c9e-29894415-2f6wz"] Nov 03 00:15:20.791833 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:20.791795 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf9431d-e50a-4d0c-83c4-015cbad7beeb" path="/var/lib/kubelet/pods/3bf9431d-e50a-4d0c-83c4-015cbad7beeb/volumes" Nov 03 00:15:20.792258 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:20.792182 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7430ed00-5e37-4edb-96a5-d43ab703ea71" path="/var/lib/kubelet/pods/7430ed00-5e37-4edb-96a5-d43ab703ea71/volumes" Nov 03 00:15:37.582613 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:15:37.582571 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:15:47.137999 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:47.137967 5114 scope.go:122] "RemoveContainer" containerID="203cc0290ee328d82006c6dbddac04f889c4907cd4e96704a984676b9b424c5b" Nov 03 00:15:47.146854 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:15:47.146831 5114 scope.go:122] "RemoveContainer" containerID="8e50e39815896cb49799a6c5aa2623ec645d25f17c9d4884dcf46ba7660595c3" Nov 03 00:16:00.454763 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:00.454723 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96"] Nov 03 00:16:06.045707 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:06.045667 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad173b80-d5866"] Nov 03 00:16:06.095231 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:06.095192 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad173b80-d5866"] Nov 03 00:16:06.791907 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:06.791868 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c779dfd9-c39c-47fc-b573-b87ac84fc3b0" path="/var/lib/kubelet/pods/c779dfd9-c39c-47fc-b573-b87ac84fc3b0/volumes" Nov 03 00:16:07.583879 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:16:07.583829 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:16:10.047884 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:10.047842 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7997331e-mhn2m"] Nov 03 00:16:10.097593 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:10.097556 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7997331e-mhn2m"] Nov 03 00:16:10.792094 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:10.792064 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66555e84-a411-4dc1-ae33-d585c63e96d2" path="/var/lib/kubelet/pods/66555e84-a411-4dc1-ae33-d585c63e96d2/volumes" Nov 03 00:16:15.255571 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:15.255528 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m"] Nov 03 00:16:15.475261 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:15.475167 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" podUID="ef6a64ff-b843-4eae-8af4-49b999bf7e15" containerName="main" containerID="cri-o://1a38a9eaf4c8514070f7f4dffb7c21f82385d33b70c4adddf41e7e04acce42af" gracePeriod=45 Nov 03 00:16:15.728757 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:15.728729 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:16:15.832331 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:15.832291 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef6a64ff-b843-4eae-8af4-49b999bf7e15-tls-certs\" (UniqueName: \"kubernetes.io/secret/ef6a64ff-b843-4eae-8af4-49b999bf7e15-tls-certs\") pod \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\" (UID: \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\") " Nov 03 00:16:15.832331 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:15.832341 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-home\" (UniqueName: \"kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-home\") pod \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\" (UID: \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\") " Nov 03 00:16:15.832662 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:15.832369 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-dshm\") pod \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\" (UID: \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\") " Nov 03 00:16:15.832662 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:15.832393 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-model-cache\") pod \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\" (UID: \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\") " Nov 03 00:16:15.832662 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:15.832456 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6a64ff-b843-4eae-8af4-49b999bf7e15-kube-api-access-lstr2\" (UniqueName: \"kubernetes.io/projected/ef6a64ff-b843-4eae-8af4-49b999bf7e15-kube-api-access-lstr2\") pod \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\" (UID: \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\") " Nov 03 00:16:15.832662 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:15.832530 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-tmp-dir\") pod \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\" (UID: \"ef6a64ff-b843-4eae-8af4-49b999bf7e15\") " Nov 03 00:16:15.832881 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:15.832685 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-home" pod "ef6a64ff-b843-4eae-8af4-49b999bf7e15" (UID: "ef6a64ff-b843-4eae-8af4-49b999bf7e15"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:15.832881 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:15.832695 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-model-cache" pod "ef6a64ff-b843-4eae-8af4-49b999bf7e15" (UID: "ef6a64ff-b843-4eae-8af4-49b999bf7e15"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:15.832881 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:15.832794 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:15.832881 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:15.832814 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:15.833033 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:15.832887 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-tmp-dir" pod "ef6a64ff-b843-4eae-8af4-49b999bf7e15" (UID: "ef6a64ff-b843-4eae-8af4-49b999bf7e15"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:15.834956 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:15.834931 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-dshm" pod "ef6a64ff-b843-4eae-8af4-49b999bf7e15" (UID: "ef6a64ff-b843-4eae-8af4-49b999bf7e15"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:15.834956 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:15.834931 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef6a64ff-b843-4eae-8af4-49b999bf7e15-tls-certs" pod "ef6a64ff-b843-4eae-8af4-49b999bf7e15" (UID: "ef6a64ff-b843-4eae-8af4-49b999bf7e15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:16:15.835105 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:15.834977 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6a64ff-b843-4eae-8af4-49b999bf7e15-kube-api-access-lstr2" pod "ef6a64ff-b843-4eae-8af4-49b999bf7e15" (UID: "ef6a64ff-b843-4eae-8af4-49b999bf7e15"). InnerVolumeSpecName "kube-api-access-lstr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:15.934151 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:15.934058 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:15.934151 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:15.934093 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lstr2\" (UniqueName: \"kubernetes.io/projected/ef6a64ff-b843-4eae-8af4-49b999bf7e15-kube-api-access-lstr2\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:15.934151 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:15.934105 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef6a64ff-b843-4eae-8af4-49b999bf7e15-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:15.934151 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:15.934117 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef6a64ff-b843-4eae-8af4-49b999bf7e15-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:16.475590 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:16.475535 5114 generic.go:356] "Generic (PLEG): container finished" podID="ef6a64ff-b843-4eae-8af4-49b999bf7e15" containerID="1a38a9eaf4c8514070f7f4dffb7c21f82385d33b70c4adddf41e7e04acce42af" exitCode=0 Nov 03 00:16:16.475590 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:16.475589 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" Nov 03 00:16:16.476150 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:16.475597 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" event={"ID":"ef6a64ff-b843-4eae-8af4-49b999bf7e15","Type":"ContainerDied","Data":"1a38a9eaf4c8514070f7f4dffb7c21f82385d33b70c4adddf41e7e04acce42af"} Nov 03 00:16:16.476150 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:16.475648 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96" event={"ID":"ef6a64ff-b843-4eae-8af4-49b999bf7e15","Type":"ContainerDied","Data":"afdb0df2ab51029eb8d0e596475a02aa09b24bfb7f973b8860c41b930817a561"} Nov 03 00:16:16.476150 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:16.475665 5114 scope.go:122] "RemoveContainer" containerID="1a38a9eaf4c8514070f7f4dffb7c21f82385d33b70c4adddf41e7e04acce42af" Nov 03 00:16:16.484870 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:16.484849 5114 scope.go:122] "RemoveContainer" containerID="1a38a9eaf4c8514070f7f4dffb7c21f82385d33b70c4adddf41e7e04acce42af" Nov 03 00:16:16.485191 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:16:16.485172 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a38a9eaf4c8514070f7f4dffb7c21f82385d33b70c4adddf41e7e04acce42af\": container with ID starting with 1a38a9eaf4c8514070f7f4dffb7c21f82385d33b70c4adddf41e7e04acce42af not found: ID does not exist" containerID="1a38a9eaf4c8514070f7f4dffb7c21f82385d33b70c4adddf41e7e04acce42af" Nov 03 00:16:16.485273 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:16.485201 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a38a9eaf4c8514070f7f4dffb7c21f82385d33b70c4adddf41e7e04acce42af"} err="failed to get container status \"1a38a9eaf4c8514070f7f4dffb7c21f82385d33b70c4adddf41e7e04acce42af\": rpc error: code = NotFound desc = could not find container \"1a38a9eaf4c8514070f7f4dffb7c21f82385d33b70c4adddf41e7e04acce42af\": container with ID starting with 1a38a9eaf4c8514070f7f4dffb7c21f82385d33b70c4adddf41e7e04acce42af not found: ID does not exist" Nov 03 00:16:16.645622 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:16.645588 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96"] Nov 03 00:16:16.694761 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:16.694724 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3bee1b/sub-test-model-3bee1b-kserve-b857999cc-6dq96"] Nov 03 00:16:16.791761 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:16.791729 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef6a64ff-b843-4eae-8af4-49b999bf7e15" path="/var/lib/kubelet/pods/ef6a64ff-b843-4eae-8af4-49b999bf7e15/volumes" Nov 03 00:16:17.046311 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:17.046223 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5616f809-l5dnn"] Nov 03 00:16:17.048986 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:17.048953 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5616f809-l5dnn"] Nov 03 00:16:18.791692 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:18.791661 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee486b7f-543e-4475-840c-103833c83b2b" path="/var/lib/kubelet/pods/ee486b7f-543e-4475-840c-103833c83b2b/volumes" Nov 03 00:16:21.046096 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:21.046057 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9948ecf8-wg2ls"] Nov 03 00:16:21.095389 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:21.095352 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9948ecf8-wg2ls"] Nov 03 00:16:22.792256 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:22.792219 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e916d42-6ee5-41dd-a6c6-c331286e11d3" path="/var/lib/kubelet/pods/2e916d42-6ee5-41dd-a6c6-c331286e11d3/volumes" Nov 03 00:16:29.047632 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:29.047592 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d41e706a-sznhb"] Nov 03 00:16:29.096923 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:29.096886 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d41e706a-sznhb"] Nov 03 00:16:30.275783 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.275744 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" podUID="3e31a11c-4ea9-40c5-b4b8-edc638c18523" containerName="main" containerID="cri-o://cdb74d2e9f206f09f05b27981b48f48cacec2bde59f40c78bf0811443d04f1ee" gracePeriod=45 Nov 03 00:16:30.517506 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.517448 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:16:30.538728 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.538685 5114 generic.go:356] "Generic (PLEG): container finished" podID="3e31a11c-4ea9-40c5-b4b8-edc638c18523" containerID="cdb74d2e9f206f09f05b27981b48f48cacec2bde59f40c78bf0811443d04f1ee" exitCode=0 Nov 03 00:16:30.538728 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.538731 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" Nov 03 00:16:30.538990 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.538728 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" event={"ID":"3e31a11c-4ea9-40c5-b4b8-edc638c18523","Type":"ContainerDied","Data":"cdb74d2e9f206f09f05b27981b48f48cacec2bde59f40c78bf0811443d04f1ee"} Nov 03 00:16:30.538990 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.538775 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m" event={"ID":"3e31a11c-4ea9-40c5-b4b8-edc638c18523","Type":"ContainerDied","Data":"37a0656a7703ed6e29ffb8f709094234daec6a0440d6169f04e551066a8c39ff"} Nov 03 00:16:30.538990 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.538788 5114 scope.go:122] "RemoveContainer" containerID="cdb74d2e9f206f09f05b27981b48f48cacec2bde59f40c78bf0811443d04f1ee" Nov 03 00:16:30.547350 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.547325 5114 scope.go:122] "RemoveContainer" containerID="cdb74d2e9f206f09f05b27981b48f48cacec2bde59f40c78bf0811443d04f1ee" Nov 03 00:16:30.547736 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:16:30.547712 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdb74d2e9f206f09f05b27981b48f48cacec2bde59f40c78bf0811443d04f1ee\": container with ID starting with cdb74d2e9f206f09f05b27981b48f48cacec2bde59f40c78bf0811443d04f1ee not found: ID does not exist" containerID="cdb74d2e9f206f09f05b27981b48f48cacec2bde59f40c78bf0811443d04f1ee" Nov 03 00:16:30.547808 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.547745 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdb74d2e9f206f09f05b27981b48f48cacec2bde59f40c78bf0811443d04f1ee"} err="failed to get container status \"cdb74d2e9f206f09f05b27981b48f48cacec2bde59f40c78bf0811443d04f1ee\": rpc error: code = NotFound desc = could not find container \"cdb74d2e9f206f09f05b27981b48f48cacec2bde59f40c78bf0811443d04f1ee\": container with ID starting with cdb74d2e9f206f09f05b27981b48f48cacec2bde59f40c78bf0811443d04f1ee not found: ID does not exist" Nov 03 00:16:30.562280 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.562241 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-home\" (UniqueName: \"kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-home\") pod \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\" (UID: \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\") " Nov 03 00:16:30.562443 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.562302 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-tmp-dir\") pod \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\" (UID: \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\") " Nov 03 00:16:30.562443 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.562330 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-model-cache\") pod \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\" (UID: \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\") " Nov 03 00:16:30.562443 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.562357 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e31a11c-4ea9-40c5-b4b8-edc638c18523-kube-api-access-wdvn8\" (UniqueName: \"kubernetes.io/projected/3e31a11c-4ea9-40c5-b4b8-edc638c18523-kube-api-access-wdvn8\") pod \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\" (UID: \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\") " Nov 03 00:16:30.562443 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.562394 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-dshm\") pod \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\" (UID: \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\") " Nov 03 00:16:30.562443 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.562421 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e31a11c-4ea9-40c5-b4b8-edc638c18523-tls-certs\" (UniqueName: \"kubernetes.io/secret/3e31a11c-4ea9-40c5-b4b8-edc638c18523-tls-certs\") pod \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\" (UID: \"3e31a11c-4ea9-40c5-b4b8-edc638c18523\") " Nov 03 00:16:30.562752 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.562613 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-home" pod "3e31a11c-4ea9-40c5-b4b8-edc638c18523" (UID: "3e31a11c-4ea9-40c5-b4b8-edc638c18523"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:30.562752 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.562686 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-tmp-dir" pod "3e31a11c-4ea9-40c5-b4b8-edc638c18523" (UID: "3e31a11c-4ea9-40c5-b4b8-edc638c18523"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:30.562752 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.562715 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-model-cache" pod "3e31a11c-4ea9-40c5-b4b8-edc638c18523" (UID: "3e31a11c-4ea9-40c5-b4b8-edc638c18523"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:30.562866 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.562771 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:30.562866 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.562803 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:30.564982 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.564944 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-dshm" pod "3e31a11c-4ea9-40c5-b4b8-edc638c18523" (UID: "3e31a11c-4ea9-40c5-b4b8-edc638c18523"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:30.565140 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.565012 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e31a11c-4ea9-40c5-b4b8-edc638c18523-tls-certs" pod "3e31a11c-4ea9-40c5-b4b8-edc638c18523" (UID: "3e31a11c-4ea9-40c5-b4b8-edc638c18523"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:16:30.565140 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.565005 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e31a11c-4ea9-40c5-b4b8-edc638c18523-kube-api-access-wdvn8" pod "3e31a11c-4ea9-40c5-b4b8-edc638c18523" (UID: "3e31a11c-4ea9-40c5-b4b8-edc638c18523"). InnerVolumeSpecName "kube-api-access-wdvn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:30.664037 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.663995 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:30.664037 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.664035 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdvn8\" (UniqueName: \"kubernetes.io/projected/3e31a11c-4ea9-40c5-b4b8-edc638c18523-kube-api-access-wdvn8\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:30.664267 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.664050 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e31a11c-4ea9-40c5-b4b8-edc638c18523-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:30.664267 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.664063 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e31a11c-4ea9-40c5-b4b8-edc638c18523-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:30.793019 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.792915 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78db8bc7-0c5b-4a63-82ed-b6d8e76ee077" path="/var/lib/kubelet/pods/78db8bc7-0c5b-4a63-82ed-b6d8e76ee077/volumes" Nov 03 00:16:30.951289 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.951249 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m"] Nov 03 00:16:30.956666 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:30.956635 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9b6c9e/sub-test-model-9b6c9e-kserve-cffd7f77c-tbx7m"] Nov 03 00:16:32.791762 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:32.791723 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e31a11c-4ea9-40c5-b4b8-edc638c18523" path="/var/lib/kubelet/pods/3e31a11c-4ea9-40c5-b4b8-edc638c18523/volumes" Nov 03 00:16:37.592957 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:16:37.592910 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:16:40.044318 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:40.044279 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c3fafca-mgsqv"] Nov 03 00:16:40.094100 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:40.094059 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c3fafca-mgsqv"] Nov 03 00:16:40.792192 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:16:40.792155 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be2c30af-068f-4165-9f57-567a24590206" path="/var/lib/kubelet/pods/be2c30af-068f-4165-9f57-567a24590206/volumes" Nov 03 00:17:07.583091 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:17:07.583052 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:17:10.460733 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:10.460687 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr"] Nov 03 00:17:10.509833 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:10.509799 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6"] Nov 03 00:17:24.356102 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.356013 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v"] Nov 03 00:17:24.356637 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.356372 5114 cpu_manager.go:414] "containerMap: removing container" podUID="3bf9431d-e50a-4d0c-83c4-015cbad7beeb" containerName="cleanup" Nov 03 00:17:24.356637 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.356387 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf9431d-e50a-4d0c-83c4-015cbad7beeb" containerName="cleanup" Nov 03 00:17:24.356637 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.356400 5114 cpu_manager.go:414] "containerMap: removing container" podUID="7430ed00-5e37-4edb-96a5-d43ab703ea71" containerName="cleanup" Nov 03 00:17:24.356637 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.356405 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7430ed00-5e37-4edb-96a5-d43ab703ea71" containerName="cleanup" Nov 03 00:17:24.356637 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.356415 5114 cpu_manager.go:414] "containerMap: removing container" podUID="ef6a64ff-b843-4eae-8af4-49b999bf7e15" containerName="main" Nov 03 00:17:24.356637 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.356420 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6a64ff-b843-4eae-8af4-49b999bf7e15" containerName="main" Nov 03 00:17:24.356637 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.356438 5114 cpu_manager.go:414] "containerMap: removing container" podUID="3e31a11c-4ea9-40c5-b4b8-edc638c18523" containerName="main" Nov 03 00:17:24.356637 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.356442 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e31a11c-4ea9-40c5-b4b8-edc638c18523" containerName="main" Nov 03 00:17:24.356637 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.356451 5114 cpu_manager.go:414] "containerMap: removing container" podUID="7430ed00-5e37-4edb-96a5-d43ab703ea71" containerName="cleanup" Nov 03 00:17:24.356637 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.356455 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7430ed00-5e37-4edb-96a5-d43ab703ea71" containerName="cleanup" Nov 03 00:17:24.356637 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.356466 5114 cpu_manager.go:414] "containerMap: removing container" podUID="3bf9431d-e50a-4d0c-83c4-015cbad7beeb" containerName="cleanup" Nov 03 00:17:24.356637 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.356471 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf9431d-e50a-4d0c-83c4-015cbad7beeb" containerName="cleanup" Nov 03 00:17:24.356637 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.356543 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="7430ed00-5e37-4edb-96a5-d43ab703ea71" containerName="cleanup" Nov 03 00:17:24.356637 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.356551 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6a64ff-b843-4eae-8af4-49b999bf7e15" containerName="main" Nov 03 00:17:24.356637 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.356560 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e31a11c-4ea9-40c5-b4b8-edc638c18523" containerName="main" Nov 03 00:17:24.356637 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.356566 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="7430ed00-5e37-4edb-96a5-d43ab703ea71" containerName="cleanup" Nov 03 00:17:24.356637 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.356572 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="7430ed00-5e37-4edb-96a5-d43ab703ea71" containerName="cleanup" Nov 03 00:17:24.356637 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.356578 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf9431d-e50a-4d0c-83c4-015cbad7beeb" containerName="cleanup" Nov 03 00:17:24.356637 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.356583 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf9431d-e50a-4d0c-83c4-015cbad7beeb" containerName="cleanup" Nov 03 00:17:24.359679 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.359652 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.411705 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.411662 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 00:17:24.429767 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.429723 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.429767 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.429770 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.430062 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.429832 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.430062 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.429854 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.430062 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.429976 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.430062 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.430016 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.430062 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.430053 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.430376 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.430146 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.430376 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.430177 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zhch\" (UniqueName: \"kubernetes.io/projected/a0604adf-bd3f-4f25-9b32-17707b58b8f7-kube-api-access-5zhch\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.461264 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.461225 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b42983-openshift-default-dockercfg-xchbk\"" Nov 03 00:17:24.462961 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.462931 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v"] Nov 03 00:17:24.531013 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.530974 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.531259 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.531028 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.531259 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.531047 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.531259 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.531075 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.531259 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.531107 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.531259 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.531125 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zhch\" (UniqueName: \"kubernetes.io/projected/a0604adf-bd3f-4f25-9b32-17707b58b8f7-kube-api-access-5zhch\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.531259 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.531153 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.531259 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.531187 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.531259 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.531225 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.531711 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.531607 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.531711 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.531643 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.531801 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.531720 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.531853 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.531833 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.531992 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.531975 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.533776 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.533755 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.534015 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.533995 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.592009 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.591979 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.592172 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.592082 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zhch\" (UniqueName: \"kubernetes.io/projected/a0604adf-bd3f-4f25-9b32-17707b58b8f7-kube-api-access-5zhch\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:24.672042 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:24.671950 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:25.100172 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.100146 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v"] Nov 03 00:17:25.102472 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:17:25.102446 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0604adf_bd3f_4f25_9b32_17707b58b8f7.slice/crio-1a1530bf376b20acfbae7d8f5ba922f56172ac3afcbdbd4327b25032d2023d3d WatchSource:0}: Error finding container 1a1530bf376b20acfbae7d8f5ba922f56172ac3afcbdbd4327b25032d2023d3d: Status 404 returned error can't find the container with id 1a1530bf376b20acfbae7d8f5ba922f56172ac3afcbdbd4327b25032d2023d3d Nov 03 00:17:25.104660 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.104621 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:17:25.104795 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.104705 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:17:25.104795 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.104749 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:17:25.482885 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.482775 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" podUID="5efc1b0f-5388-4059-b8da-5ae6c7a48501" containerName="main" containerID="cri-o://80662916c5e185f6e8096c5b78dcb677582a379f82735a7ae45c98199a866452" gracePeriod=45 Nov 03 00:17:25.530827 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.530132 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" podUID="9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6" containerName="main" containerID="cri-o://195950387f07e660041bc231096baa0853186416d1fe4c1760d21c2c02134969" gracePeriod=45 Nov 03 00:17:25.793640 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.793381 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" event={"ID":"a0604adf-bd3f-4f25-9b32-17707b58b8f7","Type":"ContainerStarted","Data":"c355355634bbde2c172deb6264bbdef85e921fdd16f7974acf44c47780b465ba"} Nov 03 00:17:25.793640 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.793444 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" event={"ID":"a0604adf-bd3f-4f25-9b32-17707b58b8f7","Type":"ContainerStarted","Data":"1a1530bf376b20acfbae7d8f5ba922f56172ac3afcbdbd4327b25032d2023d3d"} Nov 03 00:17:25.795051 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.794952 5114 generic.go:356] "Generic (PLEG): container finished" podID="9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6" containerID="195950387f07e660041bc231096baa0853186416d1fe4c1760d21c2c02134969" exitCode=0 Nov 03 00:17:25.795051 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.795029 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" event={"ID":"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6","Type":"ContainerDied","Data":"195950387f07e660041bc231096baa0853186416d1fe4c1760d21c2c02134969"} Nov 03 00:17:25.795267 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.795163 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 03 00:17:25.796272 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.796253 5114 generic.go:356] "Generic (PLEG): container finished" podID="5efc1b0f-5388-4059-b8da-5ae6c7a48501" containerID="80662916c5e185f6e8096c5b78dcb677582a379f82735a7ae45c98199a866452" exitCode=0 Nov 03 00:17:25.796366 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.796301 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" event={"ID":"5efc1b0f-5388-4059-b8da-5ae6c7a48501","Type":"ContainerDied","Data":"80662916c5e185f6e8096c5b78dcb677582a379f82735a7ae45c98199a866452"} Nov 03 00:17:25.796366 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.796337 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" event={"ID":"5efc1b0f-5388-4059-b8da-5ae6c7a48501","Type":"ContainerDied","Data":"b0e0981e60ba2c8e2cf2e4424e2109ef56fc1358ec5ccc1bfd8f38be757a8108"} Nov 03 00:17:25.796366 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.796353 5114 scope.go:122] "RemoveContainer" containerID="80662916c5e185f6e8096c5b78dcb677582a379f82735a7ae45c98199a866452" Nov 03 00:17:25.804865 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.804842 5114 scope.go:122] "RemoveContainer" containerID="80662916c5e185f6e8096c5b78dcb677582a379f82735a7ae45c98199a866452" Nov 03 00:17:25.805191 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:17:25.805167 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80662916c5e185f6e8096c5b78dcb677582a379f82735a7ae45c98199a866452\": container with ID starting with 80662916c5e185f6e8096c5b78dcb677582a379f82735a7ae45c98199a866452 not found: ID does not exist" containerID="80662916c5e185f6e8096c5b78dcb677582a379f82735a7ae45c98199a866452" Nov 03 00:17:25.805246 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.805202 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80662916c5e185f6e8096c5b78dcb677582a379f82735a7ae45c98199a866452"} err="failed to get container status \"80662916c5e185f6e8096c5b78dcb677582a379f82735a7ae45c98199a866452\": rpc error: code = NotFound desc = could not find container \"80662916c5e185f6e8096c5b78dcb677582a379f82735a7ae45c98199a866452\": container with ID starting with 80662916c5e185f6e8096c5b78dcb677582a379f82735a7ae45c98199a866452 not found: ID does not exist" Nov 03 00:17:25.844279 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.844248 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-home\" (UniqueName: \"kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-home\") pod \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\" (UID: \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\") " Nov 03 00:17:25.844422 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.844312 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5efc1b0f-5388-4059-b8da-5ae6c7a48501-kube-api-access-2wwdl\" (UniqueName: \"kubernetes.io/projected/5efc1b0f-5388-4059-b8da-5ae6c7a48501-kube-api-access-2wwdl\") pod \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\" (UID: \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\") " Nov 03 00:17:25.844422 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.844342 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5efc1b0f-5388-4059-b8da-5ae6c7a48501-tls-certs\" (UniqueName: \"kubernetes.io/secret/5efc1b0f-5388-4059-b8da-5ae6c7a48501-tls-certs\") pod \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\" (UID: \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\") " Nov 03 00:17:25.844585 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.844474 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-tmp-dir\") pod \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\" (UID: \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\") " Nov 03 00:17:25.844585 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.844534 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-dshm\") pod \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\" (UID: \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\") " Nov 03 00:17:25.844585 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.844559 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-model-cache\") pod \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\" (UID: \"5efc1b0f-5388-4059-b8da-5ae6c7a48501\") " Nov 03 00:17:25.844585 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.844564 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-home" pod "5efc1b0f-5388-4059-b8da-5ae6c7a48501" (UID: "5efc1b0f-5388-4059-b8da-5ae6c7a48501"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:25.844794 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.844767 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-tmp-dir" pod "5efc1b0f-5388-4059-b8da-5ae6c7a48501" (UID: "5efc1b0f-5388-4059-b8da-5ae6c7a48501"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:25.844985 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.844952 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:25.844985 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.844986 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:25.845304 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.845288 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-model-cache" pod "5efc1b0f-5388-4059-b8da-5ae6c7a48501" (UID: "5efc1b0f-5388-4059-b8da-5ae6c7a48501"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:25.846947 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.846922 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-dshm" pod "5efc1b0f-5388-4059-b8da-5ae6c7a48501" (UID: "5efc1b0f-5388-4059-b8da-5ae6c7a48501"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:25.847072 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.846999 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5efc1b0f-5388-4059-b8da-5ae6c7a48501-tls-certs" pod "5efc1b0f-5388-4059-b8da-5ae6c7a48501" (UID: "5efc1b0f-5388-4059-b8da-5ae6c7a48501"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:17:25.848033 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.848007 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5efc1b0f-5388-4059-b8da-5ae6c7a48501-kube-api-access-2wwdl" pod "5efc1b0f-5388-4059-b8da-5ae6c7a48501" (UID: "5efc1b0f-5388-4059-b8da-5ae6c7a48501"). InnerVolumeSpecName "kube-api-access-2wwdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:25.853125 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.853091 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 03 00:17:25.945915 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.945881 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-kube-api-access-5z8xv\" (UniqueName: \"kubernetes.io/projected/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-kube-api-access-5z8xv\") pod \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\" (UID: \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\") " Nov 03 00:17:25.945915 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.945924 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-tls-certs\") pod \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\" (UID: \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\") " Nov 03 00:17:25.946197 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.945983 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-tmp-dir\") pod \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\" (UID: \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\") " Nov 03 00:17:25.946197 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.946009 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-dshm\") pod \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\" (UID: \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\") " Nov 03 00:17:25.946197 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.946050 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-home\" (UniqueName: \"kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-home\") pod \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\" (UID: \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\") " Nov 03 00:17:25.946197 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.946082 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-model-cache\") pod \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\" (UID: \"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6\") " Nov 03 00:17:25.946431 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.946312 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-tmp-dir" pod "9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6" (UID: "9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:25.946431 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.946326 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-home" pod "9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6" (UID: "9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:25.946431 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.946373 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:25.946431 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.946394 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5efc1b0f-5388-4059-b8da-5ae6c7a48501-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:25.946431 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.946408 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wwdl\" (UniqueName: \"kubernetes.io/projected/5efc1b0f-5388-4059-b8da-5ae6c7a48501-kube-api-access-2wwdl\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:25.946431 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.946418 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5efc1b0f-5388-4059-b8da-5ae6c7a48501-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:25.946704 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.946550 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-model-cache" pod "9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6" (UID: "9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:25.948435 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.948401 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-dshm" pod "9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6" (UID: "9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:25.948576 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.948451 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-tls-certs" pod "9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6" (UID: "9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:17:25.948576 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:25.948455 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-kube-api-access-5z8xv" pod "9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6" (UID: "9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6"). InnerVolumeSpecName "kube-api-access-5z8xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:26.046828 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:26.046763 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" podStartSLOduration=2.046742277 podStartE2EDuration="2.046742277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:17:25.930903137 +0000 UTC m=+5321.283936513" watchObservedRunningTime="2026-11-03 00:17:26.046742277 +0000 UTC m=+5321.399775653" Nov 03 00:17:26.047164 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:26.047137 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:26.047228 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:26.047173 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:26.047228 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:26.047186 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:26.047228 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:26.047197 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:26.047228 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:26.047213 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z8xv\" (UniqueName: \"kubernetes.io/projected/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-kube-api-access-5z8xv\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:26.047413 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:26.047228 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:26.672963 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:26.672925 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:26.674664 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:26.674633 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 03 00:17:26.674818 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:26.674683 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" podUID="a0604adf-bd3f-4f25-9b32-17707b58b8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 03 00:17:26.792627 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:26.792587 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" podUID="9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 03 00:17:26.802267 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:26.802235 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" Nov 03 00:17:26.802434 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:26.802267 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6" event={"ID":"9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6","Type":"ContainerDied","Data":"22b41d5b4130e29e972d5e7c662a5a77ddec0ecf30f08ef455a51e798972cb36"} Nov 03 00:17:26.802434 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:26.802328 5114 scope.go:122] "RemoveContainer" containerID="195950387f07e660041bc231096baa0853186416d1fe4c1760d21c2c02134969" Nov 03 00:17:26.804074 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:26.803904 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr" Nov 03 00:17:26.971889 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:26.971844 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr"] Nov 03 00:17:26.975770 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:26.975740 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b42983/distinct-w1-b42983-kserve-7f96779c74-72hmr"] Nov 03 00:17:27.088421 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:27.088333 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6"] Nov 03 00:17:27.092443 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:27.092400 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b42983/simulator-w1-b42983-kserve-f659c5658-4cpq6"] Nov 03 00:17:27.673384 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:27.673347 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 03 00:17:27.673928 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:27.673410 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" podUID="a0604adf-bd3f-4f25-9b32-17707b58b8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 03 00:17:28.672386 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:28.672351 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 03 00:17:28.672573 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:28.672416 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" podUID="a0604adf-bd3f-4f25-9b32-17707b58b8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 03 00:17:28.792394 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:28.792361 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5efc1b0f-5388-4059-b8da-5ae6c7a48501" path="/var/lib/kubelet/pods/5efc1b0f-5388-4059-b8da-5ae6c7a48501/volumes" Nov 03 00:17:28.792824 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:28.792781 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6" path="/var/lib/kubelet/pods/9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6/volumes" Nov 03 00:17:29.672368 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:29.672330 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 03 00:17:29.672601 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:29.672392 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" podUID="a0604adf-bd3f-4f25-9b32-17707b58b8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 03 00:17:30.673067 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:30.673027 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 03 00:17:30.673628 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:30.673094 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" podUID="a0604adf-bd3f-4f25-9b32-17707b58b8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 03 00:17:31.673434 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:31.673328 5114 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 03 00:17:31.673434 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:31.673386 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" podUID="a0604adf-bd3f-4f25-9b32-17707b58b8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 03 00:17:33.575243 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:33.575188 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:33.575804 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:33.575540 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:33.576935 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:33.576913 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:37.582986 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:17:37.582952 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:17:47.199808 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:47.199776 5114 scope.go:122] "RemoveContainer" containerID="2abf2314d589c747f5abaa882754f43d2ced4cc2a440b89291c7c484cb5b9358" Nov 03 00:17:47.208716 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:47.208690 5114 scope.go:122] "RemoveContainer" containerID="7d90f8ca1f452a670dad2428c409185961ba8923a55ef1f7ad390fa65f085a0d" Nov 03 00:17:47.217952 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:47.217914 5114 scope.go:122] "RemoveContainer" containerID="d54e978a1af74f681bf60d8b78df0887a3d4ec6de274d391679fbf6670c101f4" Nov 03 00:17:47.227443 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:47.227417 5114 scope.go:122] "RemoveContainer" containerID="9f758be5783e37369cacbc8075f6c06a9620d44396349ca30a78b509810b0dd0" Nov 03 00:17:47.236337 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:47.236314 5114 scope.go:122] "RemoveContainer" containerID="0625c8d20f051bf406eb843748601f275e5593d96d69c7d0e173ec773c44447e" Nov 03 00:17:47.249102 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:47.249074 5114 scope.go:122] "RemoveContainer" containerID="a062f6d35a2dce88549a3f864e340066a065144e311e356515e9bda929d06ab2" Nov 03 00:17:52.922516 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:52.922459 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v"] Nov 03 00:17:52.923008 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:52.922687 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" podUID="a0604adf-bd3f-4f25-9b32-17707b58b8f7" containerName="istio-proxy" containerID="cri-o://c355355634bbde2c172deb6264bbdef85e921fdd16f7974acf44c47780b465ba" gracePeriod=30 Nov 03 00:17:58.175923 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.175888 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:58.336746 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.336706 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-token\" (UniqueName: \"kubernetes.io/projected/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-token\") pod \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " Nov 03 00:17:58.336948 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.336771 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-data\") pod \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " Nov 03 00:17:58.336948 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.336801 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-envoy\") pod \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " Nov 03 00:17:58.336948 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.336837 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-workload-socket\") pod \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " Nov 03 00:17:58.336948 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.336873 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0604adf-bd3f-4f25-9b32-17707b58b8f7-kube-api-access-5zhch\" (UniqueName: \"kubernetes.io/projected/a0604adf-bd3f-4f25-9b32-17707b58b8f7-kube-api-access-5zhch\") pod \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " Nov 03 00:17:58.336948 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.336895 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-credential-socket\") pod \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " Nov 03 00:17:58.336948 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.336944 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-workload-certs\") pod \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " Nov 03 00:17:58.337298 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.337000 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-podinfo\") pod \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " Nov 03 00:17:58.337298 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.337032 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istiod-ca-cert\") pod \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\" (UID: \"a0604adf-bd3f-4f25-9b32-17707b58b8f7\") " Nov 03 00:17:58.337298 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.337137 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-data" pod "a0604adf-bd3f-4f25-9b32-17707b58b8f7" (UID: "a0604adf-bd3f-4f25-9b32-17707b58b8f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:58.337298 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.337253 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-credential-socket" pod "a0604adf-bd3f-4f25-9b32-17707b58b8f7" (UID: "a0604adf-bd3f-4f25-9b32-17707b58b8f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:58.337561 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.337350 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-workload-certs" pod "a0604adf-bd3f-4f25-9b32-17707b58b8f7" (UID: "a0604adf-bd3f-4f25-9b32-17707b58b8f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:58.337561 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.337413 5114 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-data\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:58.337561 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.337435 5114 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-credential-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:58.337561 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.337416 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-workload-socket" pod "a0604adf-bd3f-4f25-9b32-17707b58b8f7" (UID: "a0604adf-bd3f-4f25-9b32-17707b58b8f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:58.337561 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.337442 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istiod-ca-cert" pod "a0604adf-bd3f-4f25-9b32-17707b58b8f7" (UID: "a0604adf-bd3f-4f25-9b32-17707b58b8f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:17:58.339507 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.339446 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-podinfo" pod "a0604adf-bd3f-4f25-9b32-17707b58b8f7" (UID: "a0604adf-bd3f-4f25-9b32-17707b58b8f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:17:58.339507 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.339461 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0604adf-bd3f-4f25-9b32-17707b58b8f7-kube-api-access-5zhch" pod "a0604adf-bd3f-4f25-9b32-17707b58b8f7" (UID: "a0604adf-bd3f-4f25-9b32-17707b58b8f7"). InnerVolumeSpecName "kube-api-access-5zhch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:58.339784 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.339759 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-envoy" pod "a0604adf-bd3f-4f25-9b32-17707b58b8f7" (UID: "a0604adf-bd3f-4f25-9b32-17707b58b8f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:58.339892 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.339798 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-token" pod "a0604adf-bd3f-4f25-9b32-17707b58b8f7" (UID: "a0604adf-bd3f-4f25-9b32-17707b58b8f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:58.438899 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.438853 5114 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-podinfo\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:58.438899 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.438894 5114 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istiod-ca-cert\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:58.438899 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.438904 5114 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-token\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:58.438899 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.438917 5114 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-istio-envoy\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:58.439218 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.438932 5114 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-workload-socket\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:58.439218 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.438940 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zhch\" (UniqueName: \"kubernetes.io/projected/a0604adf-bd3f-4f25-9b32-17707b58b8f7-kube-api-access-5zhch\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:58.439218 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.438950 5114 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0604adf-bd3f-4f25-9b32-17707b58b8f7-workload-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:58.724575 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.724462 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-d6fmn_289a684e-75c9-4b50-ad59-0ab341e55c61/manager/0.log" Nov 03 00:17:58.824057 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:17:58.824008 5114 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0604adf_bd3f_4f25_9b32_17707b58b8f7.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0604adf_bd3f_4f25_9b32_17707b58b8f7.slice/crio-1a1530bf376b20acfbae7d8f5ba922f56172ac3afcbdbd4327b25032d2023d3d\": RecentStats: unable to find data in memory cache]" Nov 03 00:17:58.970172 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.970133 5114 generic.go:356] "Generic (PLEG): container finished" podID="a0604adf-bd3f-4f25-9b32-17707b58b8f7" containerID="c355355634bbde2c172deb6264bbdef85e921fdd16f7974acf44c47780b465ba" exitCode=0 Nov 03 00:17:58.970360 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.970193 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" Nov 03 00:17:58.970360 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.970222 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" event={"ID":"a0604adf-bd3f-4f25-9b32-17707b58b8f7","Type":"ContainerDied","Data":"c355355634bbde2c172deb6264bbdef85e921fdd16f7974acf44c47780b465ba"} Nov 03 00:17:58.970360 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.970282 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v" event={"ID":"a0604adf-bd3f-4f25-9b32-17707b58b8f7","Type":"ContainerDied","Data":"1a1530bf376b20acfbae7d8f5ba922f56172ac3afcbdbd4327b25032d2023d3d"} Nov 03 00:17:58.970360 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.970305 5114 scope.go:122] "RemoveContainer" containerID="c355355634bbde2c172deb6264bbdef85e921fdd16f7974acf44c47780b465ba" Nov 03 00:17:58.978765 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.978742 5114 scope.go:122] "RemoveContainer" containerID="c355355634bbde2c172deb6264bbdef85e921fdd16f7974acf44c47780b465ba" Nov 03 00:17:58.979087 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:17:58.979066 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c355355634bbde2c172deb6264bbdef85e921fdd16f7974acf44c47780b465ba\": container with ID starting with c355355634bbde2c172deb6264bbdef85e921fdd16f7974acf44c47780b465ba not found: ID does not exist" containerID="c355355634bbde2c172deb6264bbdef85e921fdd16f7974acf44c47780b465ba" Nov 03 00:17:58.979155 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:58.979098 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c355355634bbde2c172deb6264bbdef85e921fdd16f7974acf44c47780b465ba"} err="failed to get container status \"c355355634bbde2c172deb6264bbdef85e921fdd16f7974acf44c47780b465ba\": rpc error: code = NotFound desc = could not find container \"c355355634bbde2c172deb6264bbdef85e921fdd16f7974acf44c47780b465ba\": container with ID starting with c355355634bbde2c172deb6264bbdef85e921fdd16f7974acf44c47780b465ba not found: ID does not exist" Nov 03 00:17:59.133927 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:59.133885 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v"] Nov 03 00:17:59.183557 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:59.183519 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-h6h6v"] Nov 03 00:17:59.930888 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:17:59.930853 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-bwb4h_48efe895-cacd-4880-94d4-3d965679e301/manager/0.log" Nov 03 00:18:00.791911 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:18:00.791876 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0604adf-bd3f-4f25-9b32-17707b58b8f7" path="/var/lib/kubelet/pods/a0604adf-bd3f-4f25-9b32-17707b58b8f7/volumes" Nov 03 00:18:07.583189 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:18:07.583145 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:18:37.583200 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:18:37.583160 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:19:07.583266 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:19:07.583227 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:19:09.047012 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:09.046975 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41df62c1-9ssfb"] Nov 03 00:19:09.096425 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:09.096387 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41df62c1-9ssfb"] Nov 03 00:19:10.797970 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:10.797932 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2049967-6eed-4548-b61a-ea43e34024da" path="/var/lib/kubelet/pods/d2049967-6eed-4548-b61a-ea43e34024da/volumes" Nov 03 00:19:21.045338 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:21.045291 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d5a7914-4rnj2"] Nov 03 00:19:21.094698 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:21.094658 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d5a7914-4rnj2"] Nov 03 00:19:22.792303 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:22.792256 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34a4fa4e-577f-4740-ad90-bc62fd489044" path="/var/lib/kubelet/pods/34a4fa4e-577f-4740-ad90-bc62fd489044/volumes" Nov 03 00:19:37.582896 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:19:37.582845 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:19:40.601660 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.601617 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk"] Nov 03 00:19:40.602215 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.601940 5114 cpu_manager.go:414] "containerMap: removing container" podUID="5efc1b0f-5388-4059-b8da-5ae6c7a48501" containerName="main" Nov 03 00:19:40.602215 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.601951 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5efc1b0f-5388-4059-b8da-5ae6c7a48501" containerName="main" Nov 03 00:19:40.602215 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.601969 5114 cpu_manager.go:414] "containerMap: removing container" podUID="7430ed00-5e37-4edb-96a5-d43ab703ea71" containerName="cleanup" Nov 03 00:19:40.602215 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.601975 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7430ed00-5e37-4edb-96a5-d43ab703ea71" containerName="cleanup" Nov 03 00:19:40.602215 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.601983 5114 cpu_manager.go:414] "containerMap: removing container" podUID="a0604adf-bd3f-4f25-9b32-17707b58b8f7" containerName="istio-proxy" Nov 03 00:19:40.602215 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.601989 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0604adf-bd3f-4f25-9b32-17707b58b8f7" containerName="istio-proxy" Nov 03 00:19:40.602215 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.602003 5114 cpu_manager.go:414] "containerMap: removing container" podUID="9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6" containerName="main" Nov 03 00:19:40.602215 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.602008 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6" containerName="main" Nov 03 00:19:40.602215 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.602015 5114 cpu_manager.go:414] "containerMap: removing container" podUID="3bf9431d-e50a-4d0c-83c4-015cbad7beeb" containerName="cleanup" Nov 03 00:19:40.602215 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.602022 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf9431d-e50a-4d0c-83c4-015cbad7beeb" containerName="cleanup" Nov 03 00:19:40.602215 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.602071 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0604adf-bd3f-4f25-9b32-17707b58b8f7" containerName="istio-proxy" Nov 03 00:19:40.602215 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.602081 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf9431d-e50a-4d0c-83c4-015cbad7beeb" containerName="cleanup" Nov 03 00:19:40.602215 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.602089 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b13b7b2-8504-4eb5-864e-5b6eb22cd7b6" containerName="main" Nov 03 00:19:40.602215 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.602096 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="5efc1b0f-5388-4059-b8da-5ae6c7a48501" containerName="main" Nov 03 00:19:40.604931 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.604914 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.656834 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.656794 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-98x6w\"" Nov 03 00:19:40.657042 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.656853 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 00:19:40.701636 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.701597 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.701892 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.701647 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.701892 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.701750 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.701892 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.701773 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.701892 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.701833 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kx6r\" (UniqueName: \"kubernetes.io/projected/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-kube-api-access-6kx6r\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.701892 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.701863 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.701892 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.701880 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.702256 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.701902 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.702256 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.701984 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.803689 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.803656 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.803900 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.803706 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.803900 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.803751 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.803900 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.803793 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.803900 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.803832 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.803900 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.803861 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.804137 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.803913 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kx6r\" (UniqueName: \"kubernetes.io/projected/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-kube-api-access-6kx6r\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.804137 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.803942 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.804137 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.803970 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.804266 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.804151 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.804709 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.804683 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.805083 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.805030 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk"] Nov 03 00:19:40.805275 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.805249 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.805444 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.805410 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.805723 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.805697 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.807752 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.807707 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.808175 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.808154 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.866623 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.866544 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.871203 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.871170 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kx6r\" (UniqueName: \"kubernetes.io/projected/713030b6-6b1c-4cf9-b01a-58171f6ad0c6-kube-api-access-6kx6r\") pod \"maas-default-gateway-openshift-default-694db55c87-ksnhk\" (UID: \"713030b6-6b1c-4cf9-b01a-58171f6ad0c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:40.917654 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:40.917613 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:41.342942 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:41.342875 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk"] Nov 03 00:19:41.345694 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:19:41.345667 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod713030b6_6b1c_4cf9_b01a_58171f6ad0c6.slice/crio-7094d1b5717edef1e07a6a1d6132bfde5ea70c8e3b77afccd94a92fb72786bbe WatchSource:0}: Error finding container 7094d1b5717edef1e07a6a1d6132bfde5ea70c8e3b77afccd94a92fb72786bbe: Status 404 returned error can't find the container with id 7094d1b5717edef1e07a6a1d6132bfde5ea70c8e3b77afccd94a92fb72786bbe Nov 03 00:19:41.347748 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:41.347718 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:19:41.347807 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:41.347785 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:19:41.347852 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:41.347815 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:19:41.406741 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:41.406708 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" event={"ID":"713030b6-6b1c-4cf9-b01a-58171f6ad0c6","Type":"ContainerStarted","Data":"fe23cd22de9e269665bd1b7762ffb957d70c2d4848fa8f452f60cfdb30633f62"} Nov 03 00:19:41.406878 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:41.406770 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" event={"ID":"713030b6-6b1c-4cf9-b01a-58171f6ad0c6","Type":"ContainerStarted","Data":"7094d1b5717edef1e07a6a1d6132bfde5ea70c8e3b77afccd94a92fb72786bbe"} Nov 03 00:19:41.526164 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:41.526099 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" podStartSLOduration=1.526084371 podStartE2EDuration="1.526084371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:19:41.525016028 +0000 UTC m=+5456.878049403" watchObservedRunningTime="2026-11-03 00:19:41.526084371 +0000 UTC m=+5456.879117745" Nov 03 00:19:41.929506 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:41.923606 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:42.926942 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:42.926902 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ksnhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 03 00:19:42.927264 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:42.927230 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" podUID="713030b6-6b1c-4cf9-b01a-58171f6ad0c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 03 00:19:43.923999 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:43.923955 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ksnhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 03 00:19:43.924829 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:43.924791 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" podUID="713030b6-6b1c-4cf9-b01a-58171f6ad0c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 03 00:19:44.929655 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:44.929613 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ksnhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 03 00:19:44.930335 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:44.930297 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" podUID="713030b6-6b1c-4cf9-b01a-58171f6ad0c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 03 00:19:45.925999 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:45.925953 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ksnhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 03 00:19:45.926337 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:45.926304 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" podUID="713030b6-6b1c-4cf9-b01a-58171f6ad0c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 03 00:19:46.926510 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:46.924952 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ksnhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 03 00:19:46.926510 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:46.925051 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" podUID="713030b6-6b1c-4cf9-b01a-58171f6ad0c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 03 00:19:47.918529 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:47.918465 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ksnhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 03 00:19:47.918769 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:47.918536 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" podUID="713030b6-6b1c-4cf9-b01a-58171f6ad0c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 03 00:19:48.918288 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:48.918236 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ksnhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 03 00:19:48.918927 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:48.918298 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" podUID="713030b6-6b1c-4cf9-b01a-58171f6ad0c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 03 00:19:50.959717 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:50.959677 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ksnhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:19:50.960315 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:50.959748 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" podUID="713030b6-6b1c-4cf9-b01a-58171f6ad0c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:19:51.743414 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:51.743380 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:51.743905 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:51.743762 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:19:51.744660 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:19:51.744638 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" Nov 03 00:20:07.584288 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:20:07.584247 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:20:37.586830 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:20:37.586740 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:20:41.044883 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.044848 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz"] Nov 03 00:20:41.047990 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.047969 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.094365 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.094326 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh"] Nov 03 00:20:41.097539 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.097519 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.154505 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.154446 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz"] Nov 03 00:20:41.158337 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.158305 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ba6ae1-0768-4641-9026-bce647422531-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.158522 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.158349 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ba6ae1-0768-4641-9026-bce647422531-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.158522 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.158471 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ba6ae1-0768-4641-9026-bce647422531-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.158664 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.158579 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ba6ae1-0768-4641-9026-bce647422531-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.158664 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.158651 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ba6ae1-0768-4641-9026-bce647422531-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.158783 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.158714 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ba6ae1-0768-4641-9026-bce647422531-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.158783 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.158752 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ba6ae1-0768-4641-9026-bce647422531-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.158887 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.158805 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d45tm\" (UniqueName: \"kubernetes.io/projected/19ba6ae1-0768-4641-9026-bce647422531-kube-api-access-d45tm\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.158887 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.158827 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ba6ae1-0768-4641-9026-bce647422531-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.210832 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.210792 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh"] Nov 03 00:20:41.259807 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.259752 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ba6ae1-0768-4641-9026-bce647422531-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.260015 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.259828 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e963bb-8c2f-4628-a009-ebbebd62394f-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.260015 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.259859 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ba6ae1-0768-4641-9026-bce647422531-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.260015 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.259887 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r87l\" (UniqueName: \"kubernetes.io/projected/b4e963bb-8c2f-4628-a009-ebbebd62394f-kube-api-access-9r87l\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.260015 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.259975 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e963bb-8c2f-4628-a009-ebbebd62394f-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.260163 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.260041 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e963bb-8c2f-4628-a009-ebbebd62394f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.260163 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.260084 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ba6ae1-0768-4641-9026-bce647422531-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.260163 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.260137 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ba6ae1-0768-4641-9026-bce647422531-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.260163 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.260159 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e963bb-8c2f-4628-a009-ebbebd62394f-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.260343 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.260183 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ba6ae1-0768-4641-9026-bce647422531-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.260343 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.260287 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ba6ae1-0768-4641-9026-bce647422531-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.260343 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.260327 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e963bb-8c2f-4628-a009-ebbebd62394f-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.260546 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.260352 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ba6ae1-0768-4641-9026-bce647422531-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.260546 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.260423 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ba6ae1-0768-4641-9026-bce647422531-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.260546 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.260465 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e963bb-8c2f-4628-a009-ebbebd62394f-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.260546 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.260544 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ba6ae1-0768-4641-9026-bce647422531-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.260768 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.260565 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d45tm\" (UniqueName: \"kubernetes.io/projected/19ba6ae1-0768-4641-9026-bce647422531-kube-api-access-d45tm\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.260768 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.260583 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ba6ae1-0768-4641-9026-bce647422531-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.260768 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.260594 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ba6ae1-0768-4641-9026-bce647422531-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.260768 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.260619 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e963bb-8c2f-4628-a009-ebbebd62394f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.260768 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.260659 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e963bb-8c2f-4628-a009-ebbebd62394f-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.260970 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.260923 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ba6ae1-0768-4641-9026-bce647422531-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.261186 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.261170 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ba6ae1-0768-4641-9026-bce647422531-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.262602 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.262582 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ba6ae1-0768-4641-9026-bce647422531-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.262861 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.262834 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ba6ae1-0768-4641-9026-bce647422531-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.321602 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.321512 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d45tm\" (UniqueName: \"kubernetes.io/projected/19ba6ae1-0768-4641-9026-bce647422531-kube-api-access-d45tm\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.321602 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.321528 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ba6ae1-0768-4641-9026-bce647422531-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-7zbtz\" (UID: \"19ba6ae1-0768-4641-9026-bce647422531\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.361337 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.361289 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:41.361557 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.361513 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e963bb-8c2f-4628-a009-ebbebd62394f-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.361557 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.361554 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r87l\" (UniqueName: \"kubernetes.io/projected/b4e963bb-8c2f-4628-a009-ebbebd62394f-kube-api-access-9r87l\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.361664 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.361572 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e963bb-8c2f-4628-a009-ebbebd62394f-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.361664 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.361603 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e963bb-8c2f-4628-a009-ebbebd62394f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.361664 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.361630 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e963bb-8c2f-4628-a009-ebbebd62394f-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.361827 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.361674 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e963bb-8c2f-4628-a009-ebbebd62394f-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.361827 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.361694 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e963bb-8c2f-4628-a009-ebbebd62394f-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.361827 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.361731 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e963bb-8c2f-4628-a009-ebbebd62394f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.361827 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.361768 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e963bb-8c2f-4628-a009-ebbebd62394f-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.362109 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.361925 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e963bb-8c2f-4628-a009-ebbebd62394f-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.362177 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.362148 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e963bb-8c2f-4628-a009-ebbebd62394f-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.362252 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.362228 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e963bb-8c2f-4628-a009-ebbebd62394f-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.362299 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.362248 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e963bb-8c2f-4628-a009-ebbebd62394f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.362299 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.362267 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e963bb-8c2f-4628-a009-ebbebd62394f-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.364241 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.364219 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e963bb-8c2f-4628-a009-ebbebd62394f-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.364597 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.364576 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e963bb-8c2f-4628-a009-ebbebd62394f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.424221 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.424187 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r87l\" (UniqueName: \"kubernetes.io/projected/b4e963bb-8c2f-4628-a009-ebbebd62394f-kube-api-access-9r87l\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.425076 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.425047 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e963bb-8c2f-4628-a009-ebbebd62394f-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-92jjh\" (UID: \"b4e963bb-8c2f-4628-a009-ebbebd62394f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.716566 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.716439 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:41.792354 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.791570 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz"] Nov 03 00:20:41.795061 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:20:41.795028 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ba6ae1_0768_4641_9026_bce647422531.slice/crio-c23dbb6945d1f20c294ba4b5060030389733e66ce0c52dd0797ef63a7759ded8 WatchSource:0}: Error finding container c23dbb6945d1f20c294ba4b5060030389733e66ce0c52dd0797ef63a7759ded8: Status 404 returned error can't find the container with id c23dbb6945d1f20c294ba4b5060030389733e66ce0c52dd0797ef63a7759ded8 Nov 03 00:20:41.797339 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.797302 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:41.797454 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.797380 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:41.797454 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:41.797410 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:42.091082 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:42.091044 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh"] Nov 03 00:20:42.095049 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:20:42.095022 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e963bb_8c2f_4628_a009_ebbebd62394f.slice/crio-d9308219ed84eaf5f842911d5cf3542e17682930fa2d185ea741bac311dc3f48 WatchSource:0}: Error finding container d9308219ed84eaf5f842911d5cf3542e17682930fa2d185ea741bac311dc3f48: Status 404 returned error can't find the container with id d9308219ed84eaf5f842911d5cf3542e17682930fa2d185ea741bac311dc3f48 Nov 03 00:20:42.097527 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:42.097464 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:42.097619 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:42.097585 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:42.097660 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:42.097630 5114 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:42.693592 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:42.693551 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" event={"ID":"19ba6ae1-0768-4641-9026-bce647422531","Type":"ContainerStarted","Data":"f492c964a5915128345784de469acf3f0e1a6d19ec3206f3e158689311b64cf3"} Nov 03 00:20:42.693815 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:42.693614 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" event={"ID":"19ba6ae1-0768-4641-9026-bce647422531","Type":"ContainerStarted","Data":"c23dbb6945d1f20c294ba4b5060030389733e66ce0c52dd0797ef63a7759ded8"} Nov 03 00:20:42.695205 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:42.695173 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" event={"ID":"b4e963bb-8c2f-4628-a009-ebbebd62394f","Type":"ContainerStarted","Data":"074eb5eebde94d35069d8a42bb729be121bae7d2b31a29e09b11c2daa3c86dbc"} Nov 03 00:20:42.695352 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:42.695221 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" event={"ID":"b4e963bb-8c2f-4628-a009-ebbebd62394f","Type":"ContainerStarted","Data":"d9308219ed84eaf5f842911d5cf3542e17682930fa2d185ea741bac311dc3f48"} Nov 03 00:20:42.723508 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:42.719609 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:42.939754 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:42.939672 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" podStartSLOduration=1.939646669 podStartE2EDuration="1.939646669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:42.818056217 +0000 UTC m=+5518.171089633" watchObservedRunningTime="2026-11-03 00:20:42.939646669 +0000 UTC m=+5518.292680044" Nov 03 00:20:43.361884 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:43.361845 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:43.363012 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:43.362994 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7zbtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Nov 03 00:20:43.363090 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:43.363031 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" podUID="19ba6ae1-0768-4641-9026-bce647422531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Nov 03 00:20:43.726741 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:43.726630 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-92jjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 03 00:20:43.727037 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:43.727006 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" podUID="b4e963bb-8c2f-4628-a009-ebbebd62394f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 03 00:20:44.362499 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:44.362454 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7zbtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Nov 03 00:20:44.362940 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:44.362537 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" podUID="19ba6ae1-0768-4641-9026-bce647422531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Nov 03 00:20:44.729574 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:44.726803 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-92jjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 03 00:20:44.729574 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:44.726869 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" podUID="b4e963bb-8c2f-4628-a009-ebbebd62394f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 03 00:20:45.361928 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:45.361886 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7zbtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Nov 03 00:20:45.362129 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:45.361951 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" podUID="19ba6ae1-0768-4641-9026-bce647422531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Nov 03 00:20:45.723572 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:45.722628 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-92jjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 03 00:20:45.723572 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:45.722710 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" podUID="b4e963bb-8c2f-4628-a009-ebbebd62394f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 03 00:20:46.362511 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:46.362180 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7zbtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Nov 03 00:20:46.362511 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:46.362251 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" podUID="19ba6ae1-0768-4641-9026-bce647422531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Nov 03 00:20:46.722571 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:46.720017 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-92jjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 03 00:20:46.722571 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:46.720083 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" podUID="b4e963bb-8c2f-4628-a009-ebbebd62394f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 03 00:20:47.332570 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:47.332531 5114 scope.go:122] "RemoveContainer" containerID="4421e9acbc91c4be78692eb864855d7662d4e2377a78488200b6b62f07659652" Nov 03 00:20:47.372903 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:47.372875 5114 scope.go:122] "RemoveContainer" containerID="43fe4b1903702ba7a5a488eeb5018ba33586c0810f8ef881a7a602d4937488b1" Nov 03 00:20:47.376976 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:47.376812 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7zbtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Nov 03 00:20:47.377272 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:47.377241 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" podUID="19ba6ae1-0768-4641-9026-bce647422531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Nov 03 00:20:47.718040 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:47.717936 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-92jjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 03 00:20:47.718383 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:47.718348 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" podUID="b4e963bb-8c2f-4628-a009-ebbebd62394f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 03 00:20:48.362107 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:48.362070 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7zbtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Nov 03 00:20:48.362577 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:48.362126 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" podUID="19ba6ae1-0768-4641-9026-bce647422531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Nov 03 00:20:48.717187 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:48.717082 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-92jjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 03 00:20:48.717187 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:48.717150 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" podUID="b4e963bb-8c2f-4628-a009-ebbebd62394f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 03 00:20:49.361892 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:49.361847 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7zbtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Nov 03 00:20:49.362092 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:49.361921 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" podUID="19ba6ae1-0768-4641-9026-bce647422531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Nov 03 00:20:49.717397 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:49.717287 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-92jjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 03 00:20:49.717397 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:49.717368 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" podUID="b4e963bb-8c2f-4628-a009-ebbebd62394f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 03 00:20:51.402673 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:51.402635 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7zbtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:20:51.403120 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:51.402686 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" podUID="19ba6ae1-0768-4641-9026-bce647422531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:20:51.759419 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:51.758688 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-92jjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:20:51.759419 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:51.758765 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" podUID="b4e963bb-8c2f-4628-a009-ebbebd62394f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:20:52.444736 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:52.444670 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7zbtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:20:52.445322 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:52.444784 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" podUID="19ba6ae1-0768-4641-9026-bce647422531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:20:52.657278 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:52.657244 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:52.657601 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:52.657539 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:52.658393 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:52.658372 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" Nov 03 00:20:52.800676 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:52.800636 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-92jjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:20:52.800902 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:52.800693 5114 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" podUID="b4e963bb-8c2f-4628-a009-ebbebd62394f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:20:52.828358 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:52.828287 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" podStartSLOduration=11.828264207 podStartE2EDuration="11.828264207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:42.940381673 +0000 UTC m=+5518.293415048" watchObservedRunningTime="2026-11-03 00:20:52.828264207 +0000 UTC m=+5528.181297583" Nov 03 00:20:52.875306 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:52.875264 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:52.875819 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:52.875795 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:20:52.940787 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:20:52.940751 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" Nov 03 00:21:07.583145 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:21:07.583103 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:21:37.583282 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:21:37.583242 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:21:47.343060 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:47.343011 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j"] Nov 03 00:21:47.347169 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:47.347134 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:21:47.449682 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:47.449628 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-37a87c71-kserve-self-signed-certs\"" Nov 03 00:21:47.452001 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:47.451968 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j"] Nov 03 00:21:47.462039 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:47.461991 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgknq\" (UniqueName: \"kubernetes.io/projected/a0d98204-f724-48c3-b84e-17bc22eecb2f-kube-api-access-kgknq\") pod \"e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j\" (UID: \"a0d98204-f724-48c3-b84e-17bc22eecb2f\") " pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:21:47.462272 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:47.462057 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d98204-f724-48c3-b84e-17bc22eecb2f-tls-certs\") pod \"e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j\" (UID: \"a0d98204-f724-48c3-b84e-17bc22eecb2f\") " pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:21:47.462272 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:47.462095 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-model-cache\") pod \"e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j\" (UID: \"a0d98204-f724-48c3-b84e-17bc22eecb2f\") " pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:21:47.462272 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:47.462146 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-tmp-dir\") pod \"e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j\" (UID: \"a0d98204-f724-48c3-b84e-17bc22eecb2f\") " pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:21:47.462272 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:47.462183 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-dshm\") pod \"e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j\" (UID: \"a0d98204-f724-48c3-b84e-17bc22eecb2f\") " pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:21:47.462272 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:47.462219 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-home\") pod \"e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j\" (UID: \"a0d98204-f724-48c3-b84e-17bc22eecb2f\") " pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:21:47.563183 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:47.563116 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgknq\" (UniqueName: \"kubernetes.io/projected/a0d98204-f724-48c3-b84e-17bc22eecb2f-kube-api-access-kgknq\") pod \"e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j\" (UID: \"a0d98204-f724-48c3-b84e-17bc22eecb2f\") " pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:21:47.564072 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:47.564041 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d98204-f724-48c3-b84e-17bc22eecb2f-tls-certs\") pod \"e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j\" (UID: \"a0d98204-f724-48c3-b84e-17bc22eecb2f\") " pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:21:47.565137 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:47.565113 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-model-cache\") pod \"e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j\" (UID: \"a0d98204-f724-48c3-b84e-17bc22eecb2f\") " pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:21:47.566040 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:47.565773 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-model-cache\") pod \"e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j\" (UID: \"a0d98204-f724-48c3-b84e-17bc22eecb2f\") " pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:21:47.566721 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:47.566477 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-tmp-dir\") pod \"e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j\" (UID: \"a0d98204-f724-48c3-b84e-17bc22eecb2f\") " pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:21:47.566991 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:47.566971 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-tmp-dir\") pod \"e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j\" (UID: \"a0d98204-f724-48c3-b84e-17bc22eecb2f\") " pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:21:47.567462 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:47.567187 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-dshm\") pod \"e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j\" (UID: \"a0d98204-f724-48c3-b84e-17bc22eecb2f\") " pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:21:47.568051 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:47.567983 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-home\") pod \"e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j\" (UID: \"a0d98204-f724-48c3-b84e-17bc22eecb2f\") " pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:21:47.568302 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:47.568284 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-home\") pod \"e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j\" (UID: \"a0d98204-f724-48c3-b84e-17bc22eecb2f\") " pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:21:47.574653 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:47.574590 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d98204-f724-48c3-b84e-17bc22eecb2f-tls-certs\") pod \"e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j\" (UID: \"a0d98204-f724-48c3-b84e-17bc22eecb2f\") " pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:21:47.575371 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:47.575347 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-dshm\") pod \"e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j\" (UID: \"a0d98204-f724-48c3-b84e-17bc22eecb2f\") " pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:21:47.626620 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:47.626465 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgknq\" (UniqueName: \"kubernetes.io/projected/a0d98204-f724-48c3-b84e-17bc22eecb2f-kube-api-access-kgknq\") pod \"e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j\" (UID: \"a0d98204-f724-48c3-b84e-17bc22eecb2f\") " pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:21:47.666112 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:47.665686 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:21:48.202668 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:48.199667 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j"] Nov 03 00:21:49.069711 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:49.069670 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" event={"ID":"a0d98204-f724-48c3-b84e-17bc22eecb2f","Type":"ContainerStarted","Data":"7a8e5d065f3b73c4c64186508814148fa2fc8747a32f05e821c828518ac95f38"} Nov 03 00:21:49.070319 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:49.070293 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" event={"ID":"a0d98204-f724-48c3-b84e-17bc22eecb2f","Type":"ContainerStarted","Data":"26b4e0ac91ebe2189c37aaa9edd685a46f8838daba3cd4184aaa18833b53b600"} Nov 03 00:21:49.186501 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:49.186434 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" podStartSLOduration=2.186420275 podStartE2EDuration="2.186420275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:49.185220669 +0000 UTC m=+5584.538254045" watchObservedRunningTime="2026-11-03 00:21:49.186420275 +0000 UTC m=+5584.539453644" Nov 03 00:21:49.666635 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:49.666590 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:21:57.667511 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:57.666959 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:21:57.688630 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:57.688589 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:21:58.188263 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:21:58.188202 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:22:07.583382 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:22:07.583339 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:22:37.582775 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:22:37.582724 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:22:56.178191 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:22:56.178150 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j"] Nov 03 00:23:07.584372 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:23:07.584330 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:23:11.199858 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.199813 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" podUID="a0d98204-f724-48c3-b84e-17bc22eecb2f" containerName="main" containerID="cri-o://7a8e5d065f3b73c4c64186508814148fa2fc8747a32f05e821c828518ac95f38" gracePeriod=45 Nov 03 00:23:11.445477 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.445444 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:23:11.486290 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.486198 5114 generic.go:356] "Generic (PLEG): container finished" podID="a0d98204-f724-48c3-b84e-17bc22eecb2f" containerID="7a8e5d065f3b73c4c64186508814148fa2fc8747a32f05e821c828518ac95f38" exitCode=0 Nov 03 00:23:11.486290 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.486248 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" Nov 03 00:23:11.486538 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.486281 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" event={"ID":"a0d98204-f724-48c3-b84e-17bc22eecb2f","Type":"ContainerDied","Data":"7a8e5d065f3b73c4c64186508814148fa2fc8747a32f05e821c828518ac95f38"} Nov 03 00:23:11.486538 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.486325 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j" event={"ID":"a0d98204-f724-48c3-b84e-17bc22eecb2f","Type":"ContainerDied","Data":"26b4e0ac91ebe2189c37aaa9edd685a46f8838daba3cd4184aaa18833b53b600"} Nov 03 00:23:11.486538 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.486342 5114 scope.go:122] "RemoveContainer" containerID="7a8e5d065f3b73c4c64186508814148fa2fc8747a32f05e821c828518ac95f38" Nov 03 00:23:11.494906 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.494882 5114 scope.go:122] "RemoveContainer" containerID="7a8e5d065f3b73c4c64186508814148fa2fc8747a32f05e821c828518ac95f38" Nov 03 00:23:11.495244 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:23:11.495222 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a8e5d065f3b73c4c64186508814148fa2fc8747a32f05e821c828518ac95f38\": container with ID starting with 7a8e5d065f3b73c4c64186508814148fa2fc8747a32f05e821c828518ac95f38 not found: ID does not exist" containerID="7a8e5d065f3b73c4c64186508814148fa2fc8747a32f05e821c828518ac95f38" Nov 03 00:23:11.495297 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.495257 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a8e5d065f3b73c4c64186508814148fa2fc8747a32f05e821c828518ac95f38"} err="failed to get container status \"7a8e5d065f3b73c4c64186508814148fa2fc8747a32f05e821c828518ac95f38\": rpc error: code = NotFound desc = could not find container \"7a8e5d065f3b73c4c64186508814148fa2fc8747a32f05e821c828518ac95f38\": container with ID starting with 7a8e5d065f3b73c4c64186508814148fa2fc8747a32f05e821c828518ac95f38 not found: ID does not exist" Nov 03 00:23:11.633522 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.633446 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-home\" (UniqueName: \"kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-home\") pod \"a0d98204-f724-48c3-b84e-17bc22eecb2f\" (UID: \"a0d98204-f724-48c3-b84e-17bc22eecb2f\") " Nov 03 00:23:11.633751 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.633550 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-model-cache\") pod \"a0d98204-f724-48c3-b84e-17bc22eecb2f\" (UID: \"a0d98204-f724-48c3-b84e-17bc22eecb2f\") " Nov 03 00:23:11.633751 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.633664 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d98204-f724-48c3-b84e-17bc22eecb2f-kube-api-access-kgknq\" (UniqueName: \"kubernetes.io/projected/a0d98204-f724-48c3-b84e-17bc22eecb2f-kube-api-access-kgknq\") pod \"a0d98204-f724-48c3-b84e-17bc22eecb2f\" (UID: \"a0d98204-f724-48c3-b84e-17bc22eecb2f\") " Nov 03 00:23:11.633751 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.633703 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-tmp-dir\") pod \"a0d98204-f724-48c3-b84e-17bc22eecb2f\" (UID: \"a0d98204-f724-48c3-b84e-17bc22eecb2f\") " Nov 03 00:23:11.633751 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.633742 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0d98204-f724-48c3-b84e-17bc22eecb2f-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d98204-f724-48c3-b84e-17bc22eecb2f-tls-certs\") pod \"a0d98204-f724-48c3-b84e-17bc22eecb2f\" (UID: \"a0d98204-f724-48c3-b84e-17bc22eecb2f\") " Nov 03 00:23:11.634000 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.633745 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-home" pod "a0d98204-f724-48c3-b84e-17bc22eecb2f" (UID: "a0d98204-f724-48c3-b84e-17bc22eecb2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:23:11.634000 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.633778 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-dshm\") pod \"a0d98204-f724-48c3-b84e-17bc22eecb2f\" (UID: \"a0d98204-f724-48c3-b84e-17bc22eecb2f\") " Nov 03 00:23:11.634000 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.633806 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-model-cache" pod "a0d98204-f724-48c3-b84e-17bc22eecb2f" (UID: "a0d98204-f724-48c3-b84e-17bc22eecb2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:23:11.634000 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.633981 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-tmp-dir" pod "a0d98204-f724-48c3-b84e-17bc22eecb2f" (UID: "a0d98204-f724-48c3-b84e-17bc22eecb2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:23:11.634199 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.634176 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:11.634238 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.634205 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:11.634238 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.634219 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:11.636227 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.636203 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-dshm" pod "a0d98204-f724-48c3-b84e-17bc22eecb2f" (UID: "a0d98204-f724-48c3-b84e-17bc22eecb2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:23:11.636346 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.636241 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d98204-f724-48c3-b84e-17bc22eecb2f-tls-certs" pod "a0d98204-f724-48c3-b84e-17bc22eecb2f" (UID: "a0d98204-f724-48c3-b84e-17bc22eecb2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:23:11.636346 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.636283 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d98204-f724-48c3-b84e-17bc22eecb2f-kube-api-access-kgknq" pod "a0d98204-f724-48c3-b84e-17bc22eecb2f" (UID: "a0d98204-f724-48c3-b84e-17bc22eecb2f"). InnerVolumeSpecName "kube-api-access-kgknq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:23:11.735658 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.735614 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgknq\" (UniqueName: \"kubernetes.io/projected/a0d98204-f724-48c3-b84e-17bc22eecb2f-kube-api-access-kgknq\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:11.735658 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.735648 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d98204-f724-48c3-b84e-17bc22eecb2f-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:11.735658 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.735659 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d98204-f724-48c3-b84e-17bc22eecb2f-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:11.904768 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.904727 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j"] Nov 03 00:23:11.906618 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:11.906591 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-37a87c71-kserve-5f8947cbb-np52j"] Nov 03 00:23:12.792198 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:12.792162 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0d98204-f724-48c3-b84e-17bc22eecb2f" path="/var/lib/kubelet/pods/a0d98204-f724-48c3-b84e-17bc22eecb2f/volumes" Nov 03 00:23:22.961634 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:22.961591 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8"] Nov 03 00:23:22.962079 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:22.961925 5114 cpu_manager.go:414] "containerMap: removing container" podUID="a0d98204-f724-48c3-b84e-17bc22eecb2f" containerName="main" Nov 03 00:23:22.962079 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:22.961935 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d98204-f724-48c3-b84e-17bc22eecb2f" containerName="main" Nov 03 00:23:22.962079 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:22.961998 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d98204-f724-48c3-b84e-17bc22eecb2f" containerName="main" Nov 03 00:23:22.966156 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:22.966129 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8" Nov 03 00:23:23.020972 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:23.020934 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-ee0b7a\"" Nov 03 00:23:23.031641 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:23.031600 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d8qq\" (UniqueName: \"kubernetes.io/projected/3ffa11b1-572c-4144-a4d3-49c97f02f9cf-kube-api-access-8d8qq\") pod \"payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8\" (UID: \"3ffa11b1-572c-4144-a4d3-49c97f02f9cf\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8" Nov 03 00:23:23.031826 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:23.031647 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffa11b1-572c-4144-a4d3-49c97f02f9cf-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8\" (UID: \"3ffa11b1-572c-4144-a4d3-49c97f02f9cf\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8" Nov 03 00:23:23.070659 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:23.070616 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-3648c5fd-dockercfg-fqpk5\"" Nov 03 00:23:23.072027 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:23.072001 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8"] Nov 03 00:23:23.122026 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:23.121992 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79"] Nov 03 00:23:23.125455 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:23.125424 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79" Nov 03 00:23:23.132235 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:23.132150 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tcdd\" (UniqueName: \"kubernetes.io/projected/5d1f2517-284f-43c2-b9ed-4276924d4271-kube-api-access-6tcdd\") pod \"payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79\" (UID: \"5d1f2517-284f-43c2-b9ed-4276924d4271\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79" Nov 03 00:23:23.132235 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:23.132225 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d1f2517-284f-43c2-b9ed-4276924d4271-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79\" (UID: \"5d1f2517-284f-43c2-b9ed-4276924d4271\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79" Nov 03 00:23:23.132404 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:23.132370 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d8qq\" (UniqueName: \"kubernetes.io/projected/3ffa11b1-572c-4144-a4d3-49c97f02f9cf-kube-api-access-8d8qq\") pod \"payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8\" (UID: \"3ffa11b1-572c-4144-a4d3-49c97f02f9cf\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8" Nov 03 00:23:23.132469 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:23.132404 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffa11b1-572c-4144-a4d3-49c97f02f9cf-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8\" (UID: \"3ffa11b1-572c-4144-a4d3-49c97f02f9cf\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8" Nov 03 00:23:23.133078 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:23.133056 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffa11b1-572c-4144-a4d3-49c97f02f9cf-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8\" (UID: \"3ffa11b1-572c-4144-a4d3-49c97f02f9cf\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8" Nov 03 00:23:23.183357 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:23.183322 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79"] Nov 03 00:23:23.193465 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:23.193412 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d8qq\" (UniqueName: \"kubernetes.io/projected/3ffa11b1-572c-4144-a4d3-49c97f02f9cf-kube-api-access-8d8qq\") pod \"payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8\" (UID: \"3ffa11b1-572c-4144-a4d3-49c97f02f9cf\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8" Nov 03 00:23:23.233395 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:23.233287 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tcdd\" (UniqueName: \"kubernetes.io/projected/5d1f2517-284f-43c2-b9ed-4276924d4271-kube-api-access-6tcdd\") pod \"payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79\" (UID: \"5d1f2517-284f-43c2-b9ed-4276924d4271\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79" Nov 03 00:23:23.233395 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:23.233368 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d1f2517-284f-43c2-b9ed-4276924d4271-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79\" (UID: \"5d1f2517-284f-43c2-b9ed-4276924d4271\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79" Nov 03 00:23:23.234156 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:23.234135 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d1f2517-284f-43c2-b9ed-4276924d4271-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79\" (UID: \"5d1f2517-284f-43c2-b9ed-4276924d4271\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79" Nov 03 00:23:23.278926 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:23.278878 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8" Nov 03 00:23:23.297158 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:23.297127 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tcdd\" (UniqueName: \"kubernetes.io/projected/5d1f2517-284f-43c2-b9ed-4276924d4271-kube-api-access-6tcdd\") pod \"payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79\" (UID: \"5d1f2517-284f-43c2-b9ed-4276924d4271\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79" Nov 03 00:23:23.438913 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:23.438868 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79" Nov 03 00:23:23.652562 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:23.652348 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8"] Nov 03 00:23:23.655273 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:23:23.655240 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ffa11b1_572c_4144_a4d3_49c97f02f9cf.slice/crio-41364103b495b52aec3c45a633de0fc59ef1cab1160c4b9aa4a386b2246ab457 WatchSource:0}: Error finding container 41364103b495b52aec3c45a633de0fc59ef1cab1160c4b9aa4a386b2246ab457: Status 404 returned error can't find the container with id 41364103b495b52aec3c45a633de0fc59ef1cab1160c4b9aa4a386b2246ab457 Nov 03 00:23:23.810668 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:23.810633 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79"] Nov 03 00:23:23.814274 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:23:23.814241 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d1f2517_284f_43c2_b9ed_4276924d4271.slice/crio-bb067a4dd1831219a4bc3ee4dd9dfbe782c92fa8a5a601bad9d2da3b33c0b65f WatchSource:0}: Error finding container bb067a4dd1831219a4bc3ee4dd9dfbe782c92fa8a5a601bad9d2da3b33c0b65f: Status 404 returned error can't find the container with id bb067a4dd1831219a4bc3ee4dd9dfbe782c92fa8a5a601bad9d2da3b33c0b65f Nov 03 00:23:24.563021 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:24.562809 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79" event={"ID":"5d1f2517-284f-43c2-b9ed-4276924d4271","Type":"ContainerStarted","Data":"6e1d1b5133ad5db4a84bb4e91db09e84cc8f3d79ace09a6ce6a9683b9cf08aeb"} Nov 03 00:23:24.563021 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:24.562866 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79" event={"ID":"5d1f2517-284f-43c2-b9ed-4276924d4271","Type":"ContainerStarted","Data":"bb067a4dd1831219a4bc3ee4dd9dfbe782c92fa8a5a601bad9d2da3b33c0b65f"} Nov 03 00:23:24.563021 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:24.562979 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79" Nov 03 00:23:24.564368 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:24.564346 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8" event={"ID":"3ffa11b1-572c-4144-a4d3-49c97f02f9cf","Type":"ContainerStarted","Data":"dfc1ba9d99781dde74c8cba399ae61525ec81b469e03cb0d3be5fb1ef3b1f570"} Nov 03 00:23:24.564522 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:24.564388 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8" event={"ID":"3ffa11b1-572c-4144-a4d3-49c97f02f9cf","Type":"ContainerStarted","Data":"41364103b495b52aec3c45a633de0fc59ef1cab1160c4b9aa4a386b2246ab457"} Nov 03 00:23:24.564522 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:24.564413 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8" Nov 03 00:23:24.678210 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:24.678083 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79" podStartSLOduration=1.6780602770000002 podStartE2EDuration="1.678060277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:24.677122225 +0000 UTC m=+5680.030155600" watchObservedRunningTime="2026-11-03 00:23:24.678060277 +0000 UTC m=+5680.031093653" Nov 03 00:23:24.793231 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:24.793181 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8" podStartSLOduration=2.793167259 podStartE2EDuration="2.793167259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:24.791939997 +0000 UTC m=+5680.144973376" watchObservedRunningTime="2026-11-03 00:23:24.793167259 +0000 UTC m=+5680.146200633" Nov 03 00:23:35.573166 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:35.573134 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79" Nov 03 00:23:35.573598 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:35.573338 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8" Nov 03 00:23:37.584151 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:23:37.584110 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:23:47.811706 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:47.811666 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64"] Nov 03 00:23:47.816330 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:47.816302 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:23:47.918178 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:47.918135 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ee0b7a\"/\"kube-root-ca.crt\"" Nov 03 00:23:47.918178 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:47.918135 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ee0b7a\"/\"openshift-service-ca.crt\"" Nov 03 00:23:47.918178 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:47.918174 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ee0b7a\"/\"default-dockercfg-k47hg\"" Nov 03 00:23:47.918454 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:47.918309 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ee0b7a\"/\"simulator-single-main-ee0b7a-kserve-self-signed-certs\"" Nov 03 00:23:47.919737 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:47.919715 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64"] Nov 03 00:23:47.966804 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:47.966664 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-dshm\") pod \"simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64\" (UID: \"957f6b73-f537-4f5c-8c4e-451452649cea\") " pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:23:47.968021 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:47.966833 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-home\") pod \"simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64\" (UID: \"957f6b73-f537-4f5c-8c4e-451452649cea\") " pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:23:47.968021 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:47.966976 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/957f6b73-f537-4f5c-8c4e-451452649cea-tls-certs\") pod \"simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64\" (UID: \"957f6b73-f537-4f5c-8c4e-451452649cea\") " pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:23:47.968021 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:47.967076 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6s4q\" (UniqueName: \"kubernetes.io/projected/957f6b73-f537-4f5c-8c4e-451452649cea-kube-api-access-b6s4q\") pod \"simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64\" (UID: \"957f6b73-f537-4f5c-8c4e-451452649cea\") " pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:23:47.968021 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:47.967221 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-tmp-dir\") pod \"simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64\" (UID: \"957f6b73-f537-4f5c-8c4e-451452649cea\") " pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:23:47.968021 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:47.967372 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-model-cache\") pod \"simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64\" (UID: \"957f6b73-f537-4f5c-8c4e-451452649cea\") " pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:23:48.068696 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:48.068593 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-model-cache\") pod \"simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64\" (UID: \"957f6b73-f537-4f5c-8c4e-451452649cea\") " pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:23:48.068696 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:48.068643 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-dshm\") pod \"simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64\" (UID: \"957f6b73-f537-4f5c-8c4e-451452649cea\") " pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:23:48.068696 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:48.068667 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-home\") pod \"simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64\" (UID: \"957f6b73-f537-4f5c-8c4e-451452649cea\") " pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:23:48.068696 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:48.068691 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/957f6b73-f537-4f5c-8c4e-451452649cea-tls-certs\") pod \"simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64\" (UID: \"957f6b73-f537-4f5c-8c4e-451452649cea\") " pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:23:48.069060 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:48.068721 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6s4q\" (UniqueName: \"kubernetes.io/projected/957f6b73-f537-4f5c-8c4e-451452649cea-kube-api-access-b6s4q\") pod \"simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64\" (UID: \"957f6b73-f537-4f5c-8c4e-451452649cea\") " pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:23:48.069060 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:48.068745 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-tmp-dir\") pod \"simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64\" (UID: \"957f6b73-f537-4f5c-8c4e-451452649cea\") " pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:23:48.069188 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:48.069110 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-model-cache\") pod \"simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64\" (UID: \"957f6b73-f537-4f5c-8c4e-451452649cea\") " pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:23:48.069256 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:48.069201 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-home\") pod \"simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64\" (UID: \"957f6b73-f537-4f5c-8c4e-451452649cea\") " pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:23:48.069256 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:48.069236 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-tmp-dir\") pod \"simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64\" (UID: \"957f6b73-f537-4f5c-8c4e-451452649cea\") " pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:23:48.071238 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:48.071216 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-dshm\") pod \"simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64\" (UID: \"957f6b73-f537-4f5c-8c4e-451452649cea\") " pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:23:48.071530 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:48.071510 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/957f6b73-f537-4f5c-8c4e-451452649cea-tls-certs\") pod \"simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64\" (UID: \"957f6b73-f537-4f5c-8c4e-451452649cea\") " pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:23:48.130526 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:48.130462 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6s4q\" (UniqueName: \"kubernetes.io/projected/957f6b73-f537-4f5c-8c4e-451452649cea-kube-api-access-b6s4q\") pod \"simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64\" (UID: \"957f6b73-f537-4f5c-8c4e-451452649cea\") " pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:23:48.429896 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:48.429791 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:23:48.803261 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:48.803224 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64"] Nov 03 00:23:48.806295 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:23:48.806267 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod957f6b73_f537_4f5c_8c4e_451452649cea.slice/crio-d688923cc5e55479877c795f37af34ab994d6ea4575faf5cd406ccbfa43e16bd WatchSource:0}: Error finding container d688923cc5e55479877c795f37af34ab994d6ea4575faf5cd406ccbfa43e16bd: Status 404 returned error can't find the container with id d688923cc5e55479877c795f37af34ab994d6ea4575faf5cd406ccbfa43e16bd Nov 03 00:23:49.677601 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:49.677559 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" event={"ID":"957f6b73-f537-4f5c-8c4e-451452649cea","Type":"ContainerStarted","Data":"dcc18038956aa5448b806b1afe3b2dc306e0a735ea68e518b56dd0a56b1f8988"} Nov 03 00:23:49.678070 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:49.677616 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" event={"ID":"957f6b73-f537-4f5c-8c4e-451452649cea","Type":"ContainerStarted","Data":"d688923cc5e55479877c795f37af34ab994d6ea4575faf5cd406ccbfa43e16bd"} Nov 03 00:23:49.794110 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:49.794054 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" podStartSLOduration=2.794038082 podStartE2EDuration="2.794038082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:49.79315405 +0000 UTC m=+5705.146187427" watchObservedRunningTime="2026-11-03 00:23:49.794038082 +0000 UTC m=+5705.147071501" Nov 03 00:23:50.429982 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:50.429938 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:23:58.429996 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:58.429955 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:23:58.442941 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:58.442901 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:23:58.727762 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:23:58.727671 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:24:07.583300 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:24:07.583256 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:24:37.585540 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:24:37.585466 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:24:53.902770 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:24:53.902728 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64"] Nov 03 00:24:55.926231 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:24:55.926190 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8"] Nov 03 00:24:56.029105 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:24:56.029068 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79"] Nov 03 00:25:00.926843 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:00.926786 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8" podUID="3ffa11b1-572c-4144-a4d3-49c97f02f9cf" containerName="payload-processing" containerID="cri-o://dfc1ba9d99781dde74c8cba399ae61525ec81b469e03cb0d3be5fb1ef3b1f570" gracePeriod=25 Nov 03 00:25:01.030369 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:01.030325 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79" podUID="5d1f2517-284f-43c2-b9ed-4276924d4271" containerName="payload-pre-processing" containerID="cri-o://6e1d1b5133ad5db4a84bb4e91db09e84cc8f3d79ace09a6ce6a9683b9cf08aeb" gracePeriod=25 Nov 03 00:25:05.571332 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:05.571293 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8" podUID="3ffa11b1-572c-4144-a4d3-49c97f02f9cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:25:05.571802 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:05.571307 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79" podUID="5d1f2517-284f-43c2-b9ed-4276924d4271" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:25:07.583913 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:25:07.583870 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:25:08.923717 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:08.923671 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" podUID="957f6b73-f537-4f5c-8c4e-451452649cea" containerName="main" containerID="cri-o://dcc18038956aa5448b806b1afe3b2dc306e0a735ea68e518b56dd0a56b1f8988" gracePeriod=45 Nov 03 00:25:09.177404 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:09.177336 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:25:09.317294 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:09.317255 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-model-cache\") pod \"957f6b73-f537-4f5c-8c4e-451452649cea\" (UID: \"957f6b73-f537-4f5c-8c4e-451452649cea\") " Nov 03 00:25:09.317509 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:09.317322 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-tmp-dir\") pod \"957f6b73-f537-4f5c-8c4e-451452649cea\" (UID: \"957f6b73-f537-4f5c-8c4e-451452649cea\") " Nov 03 00:25:09.317509 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:09.317353 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-dshm\") pod \"957f6b73-f537-4f5c-8c4e-451452649cea\" (UID: \"957f6b73-f537-4f5c-8c4e-451452649cea\") " Nov 03 00:25:09.317509 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:09.317389 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-home\" (UniqueName: \"kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-home\") pod \"957f6b73-f537-4f5c-8c4e-451452649cea\" (UID: \"957f6b73-f537-4f5c-8c4e-451452649cea\") " Nov 03 00:25:09.317509 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:09.317454 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/957f6b73-f537-4f5c-8c4e-451452649cea-kube-api-access-b6s4q\" (UniqueName: \"kubernetes.io/projected/957f6b73-f537-4f5c-8c4e-451452649cea-kube-api-access-b6s4q\") pod \"957f6b73-f537-4f5c-8c4e-451452649cea\" (UID: \"957f6b73-f537-4f5c-8c4e-451452649cea\") " Nov 03 00:25:09.317769 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:09.317590 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/957f6b73-f537-4f5c-8c4e-451452649cea-tls-certs\" (UniqueName: \"kubernetes.io/secret/957f6b73-f537-4f5c-8c4e-451452649cea-tls-certs\") pod \"957f6b73-f537-4f5c-8c4e-451452649cea\" (UID: \"957f6b73-f537-4f5c-8c4e-451452649cea\") " Nov 03 00:25:09.317769 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:09.317592 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-model-cache" pod "957f6b73-f537-4f5c-8c4e-451452649cea" (UID: "957f6b73-f537-4f5c-8c4e-451452649cea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:25:09.317769 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:09.317679 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-tmp-dir" pod "957f6b73-f537-4f5c-8c4e-451452649cea" (UID: "957f6b73-f537-4f5c-8c4e-451452649cea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:25:09.317904 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:09.317769 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-home" pod "957f6b73-f537-4f5c-8c4e-451452649cea" (UID: "957f6b73-f537-4f5c-8c4e-451452649cea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:25:09.317904 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:09.317887 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:09.317999 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:09.317908 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:09.317999 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:09.317922 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:09.320147 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:09.320118 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/957f6b73-f537-4f5c-8c4e-451452649cea-tls-certs" pod "957f6b73-f537-4f5c-8c4e-451452649cea" (UID: "957f6b73-f537-4f5c-8c4e-451452649cea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:25:09.320294 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:09.320273 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-dshm" pod "957f6b73-f537-4f5c-8c4e-451452649cea" (UID: "957f6b73-f537-4f5c-8c4e-451452649cea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:25:09.320360 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:09.320327 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957f6b73-f537-4f5c-8c4e-451452649cea-kube-api-access-b6s4q" pod "957f6b73-f537-4f5c-8c4e-451452649cea" (UID: "957f6b73-f537-4f5c-8c4e-451452649cea"). InnerVolumeSpecName "kube-api-access-b6s4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:25:09.419039 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:09.418995 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6s4q\" (UniqueName: \"kubernetes.io/projected/957f6b73-f537-4f5c-8c4e-451452649cea-kube-api-access-b6s4q\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:09.419039 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:09.419042 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/957f6b73-f537-4f5c-8c4e-451452649cea-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:09.419341 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:09.419052 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/957f6b73-f537-4f5c-8c4e-451452649cea-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:10.025142 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:10.025104 5114 generic.go:356] "Generic (PLEG): container finished" podID="957f6b73-f537-4f5c-8c4e-451452649cea" containerID="dcc18038956aa5448b806b1afe3b2dc306e0a735ea68e518b56dd0a56b1f8988" exitCode=0 Nov 03 00:25:10.025677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:10.025172 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" Nov 03 00:25:10.025677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:10.025193 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" event={"ID":"957f6b73-f537-4f5c-8c4e-451452649cea","Type":"ContainerDied","Data":"dcc18038956aa5448b806b1afe3b2dc306e0a735ea68e518b56dd0a56b1f8988"} Nov 03 00:25:10.025677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:10.025238 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64" event={"ID":"957f6b73-f537-4f5c-8c4e-451452649cea","Type":"ContainerDied","Data":"d688923cc5e55479877c795f37af34ab994d6ea4575faf5cd406ccbfa43e16bd"} Nov 03 00:25:10.025677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:10.025256 5114 scope.go:122] "RemoveContainer" containerID="dcc18038956aa5448b806b1afe3b2dc306e0a735ea68e518b56dd0a56b1f8988" Nov 03 00:25:10.034766 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:10.034738 5114 scope.go:122] "RemoveContainer" containerID="dcc18038956aa5448b806b1afe3b2dc306e0a735ea68e518b56dd0a56b1f8988" Nov 03 00:25:10.035115 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:25:10.035089 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcc18038956aa5448b806b1afe3b2dc306e0a735ea68e518b56dd0a56b1f8988\": container with ID starting with dcc18038956aa5448b806b1afe3b2dc306e0a735ea68e518b56dd0a56b1f8988 not found: ID does not exist" containerID="dcc18038956aa5448b806b1afe3b2dc306e0a735ea68e518b56dd0a56b1f8988" Nov 03 00:25:10.035202 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:10.035129 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcc18038956aa5448b806b1afe3b2dc306e0a735ea68e518b56dd0a56b1f8988"} err="failed to get container status \"dcc18038956aa5448b806b1afe3b2dc306e0a735ea68e518b56dd0a56b1f8988\": rpc error: code = NotFound desc = could not find container \"dcc18038956aa5448b806b1afe3b2dc306e0a735ea68e518b56dd0a56b1f8988\": container with ID starting with dcc18038956aa5448b806b1afe3b2dc306e0a735ea68e518b56dd0a56b1f8988 not found: ID does not exist" Nov 03 00:25:10.187721 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:10.187684 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64"] Nov 03 00:25:10.236914 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:10.236874 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-ee0b7a/simulator-single-main-ee0b7a-kserve-868fb6f74-kwt64"] Nov 03 00:25:10.793467 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:10.793419 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="957f6b73-f537-4f5c-8c4e-451452649cea" path="/var/lib/kubelet/pods/957f6b73-f537-4f5c-8c4e-451452649cea/volumes" Nov 03 00:25:15.571516 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:15.571448 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79" podUID="5d1f2517-284f-43c2-b9ed-4276924d4271" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:25:15.571516 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:15.571468 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8" podUID="3ffa11b1-572c-4144-a4d3-49c97f02f9cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:25:21.075619 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:21.075576 5114 generic.go:356] "Generic (PLEG): container finished" podID="3ffa11b1-572c-4144-a4d3-49c97f02f9cf" containerID="dfc1ba9d99781dde74c8cba399ae61525ec81b469e03cb0d3be5fb1ef3b1f570" exitCode=0 Nov 03 00:25:21.076043 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:21.075643 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8" event={"ID":"3ffa11b1-572c-4144-a4d3-49c97f02f9cf","Type":"ContainerDied","Data":"dfc1ba9d99781dde74c8cba399ae61525ec81b469e03cb0d3be5fb1ef3b1f570"} Nov 03 00:25:21.177287 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:21.177258 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8" Nov 03 00:25:21.273935 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:21.273905 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79" Nov 03 00:25:21.323450 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:21.323411 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ffa11b1-572c-4144-a4d3-49c97f02f9cf-kube-api-access-8d8qq\" (UniqueName: \"kubernetes.io/projected/3ffa11b1-572c-4144-a4d3-49c97f02f9cf-kube-api-access-8d8qq\") pod \"3ffa11b1-572c-4144-a4d3-49c97f02f9cf\" (UID: \"3ffa11b1-572c-4144-a4d3-49c97f02f9cf\") " Nov 03 00:25:21.323707 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:21.323604 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ffa11b1-572c-4144-a4d3-49c97f02f9cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffa11b1-572c-4144-a4d3-49c97f02f9cf-plugins-config-volume\") pod \"3ffa11b1-572c-4144-a4d3-49c97f02f9cf\" (UID: \"3ffa11b1-572c-4144-a4d3-49c97f02f9cf\") " Nov 03 00:25:21.324089 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:21.324057 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ffa11b1-572c-4144-a4d3-49c97f02f9cf-plugins-config-volume" pod "3ffa11b1-572c-4144-a4d3-49c97f02f9cf" (UID: "3ffa11b1-572c-4144-a4d3-49c97f02f9cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:25:21.325798 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:21.325775 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ffa11b1-572c-4144-a4d3-49c97f02f9cf-kube-api-access-8d8qq" pod "3ffa11b1-572c-4144-a4d3-49c97f02f9cf" (UID: "3ffa11b1-572c-4144-a4d3-49c97f02f9cf"). InnerVolumeSpecName "kube-api-access-8d8qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:25:21.424686 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:21.424645 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d1f2517-284f-43c2-b9ed-4276924d4271-kube-api-access-6tcdd\" (UniqueName: \"kubernetes.io/projected/5d1f2517-284f-43c2-b9ed-4276924d4271-kube-api-access-6tcdd\") pod \"5d1f2517-284f-43c2-b9ed-4276924d4271\" (UID: \"5d1f2517-284f-43c2-b9ed-4276924d4271\") " Nov 03 00:25:21.424686 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:21.424693 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d1f2517-284f-43c2-b9ed-4276924d4271-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d1f2517-284f-43c2-b9ed-4276924d4271-plugins-config-volume\") pod \"5d1f2517-284f-43c2-b9ed-4276924d4271\" (UID: \"5d1f2517-284f-43c2-b9ed-4276924d4271\") " Nov 03 00:25:21.425045 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:21.425027 5114 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffa11b1-572c-4144-a4d3-49c97f02f9cf-plugins-config-volume\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:21.425104 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:21.425051 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d8qq\" (UniqueName: \"kubernetes.io/projected/3ffa11b1-572c-4144-a4d3-49c97f02f9cf-kube-api-access-8d8qq\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:21.425146 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:21.425128 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d1f2517-284f-43c2-b9ed-4276924d4271-plugins-config-volume" pod "5d1f2517-284f-43c2-b9ed-4276924d4271" (UID: "5d1f2517-284f-43c2-b9ed-4276924d4271"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:25:21.427086 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:21.427063 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1f2517-284f-43c2-b9ed-4276924d4271-kube-api-access-6tcdd" pod "5d1f2517-284f-43c2-b9ed-4276924d4271" (UID: "5d1f2517-284f-43c2-b9ed-4276924d4271"). InnerVolumeSpecName "kube-api-access-6tcdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:25:21.526562 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:21.526423 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tcdd\" (UniqueName: \"kubernetes.io/projected/5d1f2517-284f-43c2-b9ed-4276924d4271-kube-api-access-6tcdd\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:21.526562 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:21.526460 5114 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d1f2517-284f-43c2-b9ed-4276924d4271-plugins-config-volume\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:22.081421 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:22.081377 5114 generic.go:356] "Generic (PLEG): container finished" podID="5d1f2517-284f-43c2-b9ed-4276924d4271" containerID="6e1d1b5133ad5db4a84bb4e91db09e84cc8f3d79ace09a6ce6a9683b9cf08aeb" exitCode=0 Nov 03 00:25:22.081949 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:22.081431 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79" Nov 03 00:25:22.081949 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:22.081469 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79" event={"ID":"5d1f2517-284f-43c2-b9ed-4276924d4271","Type":"ContainerDied","Data":"6e1d1b5133ad5db4a84bb4e91db09e84cc8f3d79ace09a6ce6a9683b9cf08aeb"} Nov 03 00:25:22.081949 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:22.081557 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79" event={"ID":"5d1f2517-284f-43c2-b9ed-4276924d4271","Type":"ContainerDied","Data":"bb067a4dd1831219a4bc3ee4dd9dfbe782c92fa8a5a601bad9d2da3b33c0b65f"} Nov 03 00:25:22.081949 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:22.081588 5114 scope.go:122] "RemoveContainer" containerID="6e1d1b5133ad5db4a84bb4e91db09e84cc8f3d79ace09a6ce6a9683b9cf08aeb" Nov 03 00:25:22.082967 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:22.082904 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8" Nov 03 00:25:22.083088 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:22.082901 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8" event={"ID":"3ffa11b1-572c-4144-a4d3-49c97f02f9cf","Type":"ContainerDied","Data":"41364103b495b52aec3c45a633de0fc59ef1cab1160c4b9aa4a386b2246ab457"} Nov 03 00:25:22.091036 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:22.091012 5114 scope.go:122] "RemoveContainer" containerID="6e1d1b5133ad5db4a84bb4e91db09e84cc8f3d79ace09a6ce6a9683b9cf08aeb" Nov 03 00:25:22.091385 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:25:22.091359 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e1d1b5133ad5db4a84bb4e91db09e84cc8f3d79ace09a6ce6a9683b9cf08aeb\": container with ID starting with 6e1d1b5133ad5db4a84bb4e91db09e84cc8f3d79ace09a6ce6a9683b9cf08aeb not found: ID does not exist" containerID="6e1d1b5133ad5db4a84bb4e91db09e84cc8f3d79ace09a6ce6a9683b9cf08aeb" Nov 03 00:25:22.091432 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:22.091401 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e1d1b5133ad5db4a84bb4e91db09e84cc8f3d79ace09a6ce6a9683b9cf08aeb"} err="failed to get container status \"6e1d1b5133ad5db4a84bb4e91db09e84cc8f3d79ace09a6ce6a9683b9cf08aeb\": rpc error: code = NotFound desc = could not find container \"6e1d1b5133ad5db4a84bb4e91db09e84cc8f3d79ace09a6ce6a9683b9cf08aeb\": container with ID starting with 6e1d1b5133ad5db4a84bb4e91db09e84cc8f3d79ace09a6ce6a9683b9cf08aeb not found: ID does not exist" Nov 03 00:25:22.091432 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:22.091425 5114 scope.go:122] "RemoveContainer" containerID="dfc1ba9d99781dde74c8cba399ae61525ec81b469e03cb0d3be5fb1ef3b1f570" Nov 03 00:25:22.243864 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:22.243768 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8"] Nov 03 00:25:22.247622 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:22.247589 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ee0b7a-c8dd9977frkpn8"] Nov 03 00:25:22.356517 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:22.356462 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79"] Nov 03 00:25:22.360144 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:22.360114 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0b7a-cd5b5ksc79"] Nov 03 00:25:22.792366 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:22.792331 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ffa11b1-572c-4144-a4d3-49c97f02f9cf" path="/var/lib/kubelet/pods/3ffa11b1-572c-4144-a4d3-49c97f02f9cf/volumes" Nov 03 00:25:22.792723 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:22.792710 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d1f2517-284f-43c2-b9ed-4276924d4271" path="/var/lib/kubelet/pods/5d1f2517-284f-43c2-b9ed-4276924d4271/volumes" Nov 03 00:25:37.584852 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:25:37.584804 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:25:52.052562 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.052519 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr"] Nov 03 00:25:52.053144 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.052943 5114 cpu_manager.go:414] "containerMap: removing container" podUID="5d1f2517-284f-43c2-b9ed-4276924d4271" containerName="payload-pre-processing" Nov 03 00:25:52.053144 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.052964 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d1f2517-284f-43c2-b9ed-4276924d4271" containerName="payload-pre-processing" Nov 03 00:25:52.053144 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.053010 5114 cpu_manager.go:414] "containerMap: removing container" podUID="957f6b73-f537-4f5c-8c4e-451452649cea" containerName="main" Nov 03 00:25:52.053144 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.053026 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="957f6b73-f537-4f5c-8c4e-451452649cea" containerName="main" Nov 03 00:25:52.053144 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.053052 5114 cpu_manager.go:414] "containerMap: removing container" podUID="3ffa11b1-572c-4144-a4d3-49c97f02f9cf" containerName="payload-processing" Nov 03 00:25:52.053144 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.053057 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ffa11b1-572c-4144-a4d3-49c97f02f9cf" containerName="payload-processing" Nov 03 00:25:52.053144 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.053123 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ffa11b1-572c-4144-a4d3-49c97f02f9cf" containerName="payload-processing" Nov 03 00:25:52.053144 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.053133 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d1f2517-284f-43c2-b9ed-4276924d4271" containerName="payload-pre-processing" Nov 03 00:25:52.053144 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.053144 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="957f6b73-f537-4f5c-8c4e-451452649cea" containerName="main" Nov 03 00:25:52.057384 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.057367 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:25:52.110148 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.110109 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-home\") pod \"e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr\" (UID: \"a17b8a18-1102-48b2-98ec-5b5adf9819df\") " pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:25:52.110339 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.110160 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a17b8a18-1102-48b2-98ec-5b5adf9819df-tls-certs\") pod \"e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr\" (UID: \"a17b8a18-1102-48b2-98ec-5b5adf9819df\") " pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:25:52.110339 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.110295 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-dshm\") pod \"e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr\" (UID: \"a17b8a18-1102-48b2-98ec-5b5adf9819df\") " pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:25:52.110452 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.110339 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r59c\" (UniqueName: \"kubernetes.io/projected/a17b8a18-1102-48b2-98ec-5b5adf9819df-kube-api-access-6r59c\") pod \"e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr\" (UID: \"a17b8a18-1102-48b2-98ec-5b5adf9819df\") " pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:25:52.110452 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.110404 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-tmp-dir\") pod \"e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr\" (UID: \"a17b8a18-1102-48b2-98ec-5b5adf9819df\") " pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:25:52.110577 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.110552 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-model-cache\") pod \"e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr\" (UID: \"a17b8a18-1102-48b2-98ec-5b5adf9819df\") " pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:25:52.159615 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.159571 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-5fa7308b-kserve-self-signed-certs\"" Nov 03 00:25:52.160858 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.160829 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr"] Nov 03 00:25:52.211342 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.211294 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-model-cache\") pod \"e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr\" (UID: \"a17b8a18-1102-48b2-98ec-5b5adf9819df\") " pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:25:52.211571 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.211357 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-home\") pod \"e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr\" (UID: \"a17b8a18-1102-48b2-98ec-5b5adf9819df\") " pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:25:52.211571 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.211383 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a17b8a18-1102-48b2-98ec-5b5adf9819df-tls-certs\") pod \"e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr\" (UID: \"a17b8a18-1102-48b2-98ec-5b5adf9819df\") " pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:25:52.211571 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.211444 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-dshm\") pod \"e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr\" (UID: \"a17b8a18-1102-48b2-98ec-5b5adf9819df\") " pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:25:52.211571 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.211473 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r59c\" (UniqueName: \"kubernetes.io/projected/a17b8a18-1102-48b2-98ec-5b5adf9819df-kube-api-access-6r59c\") pod \"e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr\" (UID: \"a17b8a18-1102-48b2-98ec-5b5adf9819df\") " pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:25:52.211571 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.211542 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-tmp-dir\") pod \"e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr\" (UID: \"a17b8a18-1102-48b2-98ec-5b5adf9819df\") " pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:25:52.211872 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.211822 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-model-cache\") pod \"e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr\" (UID: \"a17b8a18-1102-48b2-98ec-5b5adf9819df\") " pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:25:52.211931 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.211904 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-home\") pod \"e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr\" (UID: \"a17b8a18-1102-48b2-98ec-5b5adf9819df\") " pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:25:52.211931 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.211921 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-tmp-dir\") pod \"e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr\" (UID: \"a17b8a18-1102-48b2-98ec-5b5adf9819df\") " pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:25:52.214092 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.214066 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-dshm\") pod \"e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr\" (UID: \"a17b8a18-1102-48b2-98ec-5b5adf9819df\") " pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:25:52.214304 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.214288 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a17b8a18-1102-48b2-98ec-5b5adf9819df-tls-certs\") pod \"e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr\" (UID: \"a17b8a18-1102-48b2-98ec-5b5adf9819df\") " pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:25:52.272712 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.272669 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r59c\" (UniqueName: \"kubernetes.io/projected/a17b8a18-1102-48b2-98ec-5b5adf9819df-kube-api-access-6r59c\") pod \"e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr\" (UID: \"a17b8a18-1102-48b2-98ec-5b5adf9819df\") " pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:25:52.370187 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.370090 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:25:52.754599 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:52.754574 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr"] Nov 03 00:25:52.757239 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:25:52.757209 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda17b8a18_1102_48b2_98ec_5b5adf9819df.slice/crio-9a136998c1129e8385abb43fa8c157402ce56a4192ea78cd2a26a85ea3e7cc73 WatchSource:0}: Error finding container 9a136998c1129e8385abb43fa8c157402ce56a4192ea78cd2a26a85ea3e7cc73: Status 404 returned error can't find the container with id 9a136998c1129e8385abb43fa8c157402ce56a4192ea78cd2a26a85ea3e7cc73 Nov 03 00:25:53.225273 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:53.225230 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" event={"ID":"a17b8a18-1102-48b2-98ec-5b5adf9819df","Type":"ContainerStarted","Data":"7b4b60934b3c9b921ae7261a758cf941815c095861b04648f5cb08c01b242384"} Nov 03 00:25:53.225273 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:53.225274 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" event={"ID":"a17b8a18-1102-48b2-98ec-5b5adf9819df","Type":"ContainerStarted","Data":"9a136998c1129e8385abb43fa8c157402ce56a4192ea78cd2a26a85ea3e7cc73"} Nov 03 00:25:53.341884 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:53.341822 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" podStartSLOduration=1.341807106 podStartE2EDuration="1.341807106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:53.340653842 +0000 UTC m=+5828.693687229" watchObservedRunningTime="2026-11-03 00:25:53.341807106 +0000 UTC m=+5828.694840481" Nov 03 00:25:53.371214 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:25:53.371163 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:26:02.371153 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:02.371113 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:26:02.384083 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:02.384051 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:26:03.303909 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:03.303874 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:26:07.584251 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:26:07.584209 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:26:23.169596 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:23.169553 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x"] Nov 03 00:26:23.172975 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:23.172959 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:26:23.224991 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:23.224961 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-5fa7308b-kserve-self-signed-certs\"" Nov 03 00:26:23.275567 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:23.275528 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x"] Nov 03 00:26:23.311763 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:23.311724 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-model-cache\") pod \"e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x\" (UID: \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\") " pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:26:23.311990 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:23.311896 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-tmp-dir\") pod \"e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x\" (UID: \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\") " pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:26:23.311990 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:23.311960 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-dshm\") pod \"e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x\" (UID: \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\") " pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:26:23.312158 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:23.312027 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jnh4\" (UniqueName: \"kubernetes.io/projected/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-kube-api-access-7jnh4\") pod \"e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x\" (UID: \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\") " pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:26:23.312158 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:23.312090 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-home\") pod \"e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x\" (UID: \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\") " pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:26:23.312158 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:23.312127 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-tls-certs\") pod \"e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x\" (UID: \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\") " pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:26:23.413448 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:23.413405 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-home\") pod \"e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x\" (UID: \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\") " pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:26:23.413448 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:23.413449 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-tls-certs\") pod \"e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x\" (UID: \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\") " pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:26:23.413757 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:23.413476 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-model-cache\") pod \"e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x\" (UID: \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\") " pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:26:23.413757 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:23.413567 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-tmp-dir\") pod \"e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x\" (UID: \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\") " pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:26:23.413757 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:23.413591 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-dshm\") pod \"e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x\" (UID: \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\") " pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:26:23.413757 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:23.413730 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jnh4\" (UniqueName: \"kubernetes.io/projected/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-kube-api-access-7jnh4\") pod \"e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x\" (UID: \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\") " pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:26:23.413992 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:23.413923 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-home\") pod \"e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x\" (UID: \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\") " pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:26:23.413992 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:23.413964 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-model-cache\") pod \"e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x\" (UID: \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\") " pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:26:23.414078 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:23.413987 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-tmp-dir\") pod \"e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x\" (UID: \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\") " pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:26:23.416064 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:23.416045 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-dshm\") pod \"e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x\" (UID: \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\") " pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:26:23.416364 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:23.416342 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-tls-certs\") pod \"e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x\" (UID: \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\") " pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:26:23.475186 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:23.475109 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jnh4\" (UniqueName: \"kubernetes.io/projected/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-kube-api-access-7jnh4\") pod \"e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x\" (UID: \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\") " pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:26:23.485945 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:23.485911 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:26:23.872139 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:23.872060 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x"] Nov 03 00:26:23.878176 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:26:23.878141 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e380280_d7c8_4e7f_9b88_fe7c50271b2a.slice/crio-6e2ca00fe0d7a708a598edbdf54f63fc39ef541e93d1ec50b63e0eeb977b75df WatchSource:0}: Error finding container 6e2ca00fe0d7a708a598edbdf54f63fc39ef541e93d1ec50b63e0eeb977b75df: Status 404 returned error can't find the container with id 6e2ca00fe0d7a708a598edbdf54f63fc39ef541e93d1ec50b63e0eeb977b75df Nov 03 00:26:24.378934 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:24.378842 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" event={"ID":"2e380280-d7c8-4e7f-9b88-fe7c50271b2a","Type":"ContainerStarted","Data":"8a57fcce7b8f49d926cbac532bacfcdd7882affc79d58b4d74018e677a654c01"} Nov 03 00:26:24.378934 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:24.378892 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" event={"ID":"2e380280-d7c8-4e7f-9b88-fe7c50271b2a","Type":"ContainerStarted","Data":"6e2ca00fe0d7a708a598edbdf54f63fc39ef541e93d1ec50b63e0eeb977b75df"} Nov 03 00:26:24.486477 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:24.486432 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:26:24.495378 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:24.495318 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" podStartSLOduration=1.495302256 podStartE2EDuration="1.495302256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:24.494021762 +0000 UTC m=+5859.847055138" watchObservedRunningTime="2026-11-03 00:26:24.495302256 +0000 UTC m=+5859.848335628" Nov 03 00:26:33.486425 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:33.486387 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:26:33.499237 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:33.499207 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:26:34.463582 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:26:34.463547 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:26:37.584396 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:26:37.584351 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:27:07.583558 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:27:07.583511 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:27:10.159763 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:10.159724 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr"] Nov 03 00:27:25.180260 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.180206 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" podUID="a17b8a18-1102-48b2-98ec-5b5adf9819df" containerName="main" containerID="cri-o://7b4b60934b3c9b921ae7261a758cf941815c095861b04648f5cb08c01b242384" gracePeriod=45 Nov 03 00:27:25.292640 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.292586 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" podUID="a17b8a18-1102-48b2-98ec-5b5adf9819df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.212:8000/ready\": dial tcp 10.134.0.212:8000: connect: connection refused" Nov 03 00:27:25.424465 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.424437 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:27:25.450345 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.450249 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-dshm\") pod \"a17b8a18-1102-48b2-98ec-5b5adf9819df\" (UID: \"a17b8a18-1102-48b2-98ec-5b5adf9819df\") " Nov 03 00:27:25.450345 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.450303 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a17b8a18-1102-48b2-98ec-5b5adf9819df-tls-certs\" (UniqueName: \"kubernetes.io/secret/a17b8a18-1102-48b2-98ec-5b5adf9819df-tls-certs\") pod \"a17b8a18-1102-48b2-98ec-5b5adf9819df\" (UID: \"a17b8a18-1102-48b2-98ec-5b5adf9819df\") " Nov 03 00:27:25.450593 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.450381 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-model-cache\") pod \"a17b8a18-1102-48b2-98ec-5b5adf9819df\" (UID: \"a17b8a18-1102-48b2-98ec-5b5adf9819df\") " Nov 03 00:27:25.450593 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.450430 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-home\" (UniqueName: \"kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-home\") pod \"a17b8a18-1102-48b2-98ec-5b5adf9819df\" (UID: \"a17b8a18-1102-48b2-98ec-5b5adf9819df\") " Nov 03 00:27:25.450593 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.450467 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-tmp-dir\") pod \"a17b8a18-1102-48b2-98ec-5b5adf9819df\" (UID: \"a17b8a18-1102-48b2-98ec-5b5adf9819df\") " Nov 03 00:27:25.450717 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.450654 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-model-cache" pod "a17b8a18-1102-48b2-98ec-5b5adf9819df" (UID: "a17b8a18-1102-48b2-98ec-5b5adf9819df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:27:25.450786 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.450719 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-home" pod "a17b8a18-1102-48b2-98ec-5b5adf9819df" (UID: "a17b8a18-1102-48b2-98ec-5b5adf9819df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:27:25.450843 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.450791 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-tmp-dir" pod "a17b8a18-1102-48b2-98ec-5b5adf9819df" (UID: "a17b8a18-1102-48b2-98ec-5b5adf9819df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:27:25.450891 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.450861 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:27:25.450891 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.450878 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:27:25.452743 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.452717 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a17b8a18-1102-48b2-98ec-5b5adf9819df-tls-certs" pod "a17b8a18-1102-48b2-98ec-5b5adf9819df" (UID: "a17b8a18-1102-48b2-98ec-5b5adf9819df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:27:25.452743 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.452737 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-dshm" pod "a17b8a18-1102-48b2-98ec-5b5adf9819df" (UID: "a17b8a18-1102-48b2-98ec-5b5adf9819df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:27:25.551438 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.551405 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a17b8a18-1102-48b2-98ec-5b5adf9819df-kube-api-access-6r59c\" (UniqueName: \"kubernetes.io/projected/a17b8a18-1102-48b2-98ec-5b5adf9819df-kube-api-access-6r59c\") pod \"a17b8a18-1102-48b2-98ec-5b5adf9819df\" (UID: \"a17b8a18-1102-48b2-98ec-5b5adf9819df\") " Nov 03 00:27:25.551672 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.551642 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:27:25.551672 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.551654 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a17b8a18-1102-48b2-98ec-5b5adf9819df-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:27:25.551672 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.551663 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a17b8a18-1102-48b2-98ec-5b5adf9819df-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:27:25.553835 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.553807 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a17b8a18-1102-48b2-98ec-5b5adf9819df-kube-api-access-6r59c" pod "a17b8a18-1102-48b2-98ec-5b5adf9819df" (UID: "a17b8a18-1102-48b2-98ec-5b5adf9819df"). InnerVolumeSpecName "kube-api-access-6r59c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:27:25.652195 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.652150 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r59c\" (UniqueName: \"kubernetes.io/projected/a17b8a18-1102-48b2-98ec-5b5adf9819df-kube-api-access-6r59c\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:27:25.729459 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.729370 5114 generic.go:356] "Generic (PLEG): container finished" podID="a17b8a18-1102-48b2-98ec-5b5adf9819df" containerID="7b4b60934b3c9b921ae7261a758cf941815c095861b04648f5cb08c01b242384" exitCode=0 Nov 03 00:27:25.729459 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.729415 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" Nov 03 00:27:25.729702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.729460 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" event={"ID":"a17b8a18-1102-48b2-98ec-5b5adf9819df","Type":"ContainerDied","Data":"7b4b60934b3c9b921ae7261a758cf941815c095861b04648f5cb08c01b242384"} Nov 03 00:27:25.729702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.729533 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr" event={"ID":"a17b8a18-1102-48b2-98ec-5b5adf9819df","Type":"ContainerDied","Data":"9a136998c1129e8385abb43fa8c157402ce56a4192ea78cd2a26a85ea3e7cc73"} Nov 03 00:27:25.729702 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.729556 5114 scope.go:122] "RemoveContainer" containerID="7b4b60934b3c9b921ae7261a758cf941815c095861b04648f5cb08c01b242384" Nov 03 00:27:25.738904 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.738879 5114 scope.go:122] "RemoveContainer" containerID="7b4b60934b3c9b921ae7261a758cf941815c095861b04648f5cb08c01b242384" Nov 03 00:27:25.739243 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:27:25.739218 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b4b60934b3c9b921ae7261a758cf941815c095861b04648f5cb08c01b242384\": container with ID starting with 7b4b60934b3c9b921ae7261a758cf941815c095861b04648f5cb08c01b242384 not found: ID does not exist" containerID="7b4b60934b3c9b921ae7261a758cf941815c095861b04648f5cb08c01b242384" Nov 03 00:27:25.739328 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.739269 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b4b60934b3c9b921ae7261a758cf941815c095861b04648f5cb08c01b242384"} err="failed to get container status \"7b4b60934b3c9b921ae7261a758cf941815c095861b04648f5cb08c01b242384\": rpc error: code = NotFound desc = could not find container \"7b4b60934b3c9b921ae7261a758cf941815c095861b04648f5cb08c01b242384\": container with ID starting with 7b4b60934b3c9b921ae7261a758cf941815c095861b04648f5cb08c01b242384 not found: ID does not exist" Nov 03 00:27:25.892195 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.892156 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr"] Nov 03 00:27:25.897755 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:25.897724 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-5fa7308b-kserve-6648d995fc-t29mr"] Nov 03 00:27:26.579290 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:26.579258 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x"] Nov 03 00:27:26.791717 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:26.791688 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a17b8a18-1102-48b2-98ec-5b5adf9819df" path="/var/lib/kubelet/pods/a17b8a18-1102-48b2-98ec-5b5adf9819df/volumes" Nov 03 00:27:37.583845 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:27:37.583797 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:27:41.599916 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:41.599848 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" podUID="2e380280-d7c8-4e7f-9b88-fe7c50271b2a" containerName="main" containerID="cri-o://8a57fcce7b8f49d926cbac532bacfcdd7882affc79d58b4d74018e677a654c01" gracePeriod=45 Nov 03 00:27:41.827093 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:41.827047 5114 generic.go:356] "Generic (PLEG): container finished" podID="2e380280-d7c8-4e7f-9b88-fe7c50271b2a" containerID="8a57fcce7b8f49d926cbac532bacfcdd7882affc79d58b4d74018e677a654c01" exitCode=0 Nov 03 00:27:41.827263 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:41.827112 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" event={"ID":"2e380280-d7c8-4e7f-9b88-fe7c50271b2a","Type":"ContainerDied","Data":"8a57fcce7b8f49d926cbac532bacfcdd7882affc79d58b4d74018e677a654c01"} Nov 03 00:27:41.844969 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:41.844941 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:27:41.941535 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:41.941398 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-model-cache\") pod \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\" (UID: \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\") " Nov 03 00:27:41.941535 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:41.941525 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-kube-api-access-7jnh4\" (UniqueName: \"kubernetes.io/projected/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-kube-api-access-7jnh4\") pod \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\" (UID: \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\") " Nov 03 00:27:41.941803 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:41.941579 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-home\" (UniqueName: \"kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-home\") pod \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\" (UID: \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\") " Nov 03 00:27:41.941803 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:41.941625 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-tls-certs\") pod \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\" (UID: \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\") " Nov 03 00:27:41.941803 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:41.941675 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-dshm\") pod \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\" (UID: \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\") " Nov 03 00:27:41.941803 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:41.941710 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-tmp-dir\") pod \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\" (UID: \"2e380280-d7c8-4e7f-9b88-fe7c50271b2a\") " Nov 03 00:27:41.941803 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:41.941748 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-model-cache" pod "2e380280-d7c8-4e7f-9b88-fe7c50271b2a" (UID: "2e380280-d7c8-4e7f-9b88-fe7c50271b2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:27:41.942519 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:41.941863 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-home" pod "2e380280-d7c8-4e7f-9b88-fe7c50271b2a" (UID: "2e380280-d7c8-4e7f-9b88-fe7c50271b2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:27:41.942519 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:41.941978 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:27:41.942519 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:41.941999 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:27:41.942519 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:41.942001 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-tmp-dir" pod "2e380280-d7c8-4e7f-9b88-fe7c50271b2a" (UID: "2e380280-d7c8-4e7f-9b88-fe7c50271b2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:27:41.943938 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:41.943914 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-kube-api-access-7jnh4" pod "2e380280-d7c8-4e7f-9b88-fe7c50271b2a" (UID: "2e380280-d7c8-4e7f-9b88-fe7c50271b2a"). InnerVolumeSpecName "kube-api-access-7jnh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:27:41.944069 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:41.944047 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-tls-certs" pod "2e380280-d7c8-4e7f-9b88-fe7c50271b2a" (UID: "2e380280-d7c8-4e7f-9b88-fe7c50271b2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:27:41.944359 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:41.944339 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-dshm" pod "2e380280-d7c8-4e7f-9b88-fe7c50271b2a" (UID: "2e380280-d7c8-4e7f-9b88-fe7c50271b2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:27:42.042755 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:42.042710 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:27:42.042755 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:42.042746 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:27:42.042755 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:42.042759 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:27:42.043027 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:42.042775 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jnh4\" (UniqueName: \"kubernetes.io/projected/2e380280-d7c8-4e7f-9b88-fe7c50271b2a-kube-api-access-7jnh4\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:27:42.832995 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:42.832953 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" event={"ID":"2e380280-d7c8-4e7f-9b88-fe7c50271b2a","Type":"ContainerDied","Data":"6e2ca00fe0d7a708a598edbdf54f63fc39ef541e93d1ec50b63e0eeb977b75df"} Nov 03 00:27:42.832995 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:42.832979 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x" Nov 03 00:27:42.833548 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:42.833019 5114 scope.go:122] "RemoveContainer" containerID="8a57fcce7b8f49d926cbac532bacfcdd7882affc79d58b4d74018e677a654c01" Nov 03 00:27:42.996868 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:42.996830 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x"] Nov 03 00:27:43.046705 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:43.046671 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-5fa7308b-kserve-7dc8f487cc-zk78x"] Nov 03 00:27:44.792140 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:27:44.792105 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e380280-d7c8-4e7f-9b88-fe7c50271b2a" path="/var/lib/kubelet/pods/2e380280-d7c8-4e7f-9b88-fe7c50271b2a/volumes" Nov 03 00:28:07.583449 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:28:07.583407 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:28:37.584537 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:28:37.584476 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:29:07.583913 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:29:07.583867 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:29:37.585133 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:29:37.585089 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:30:07.584151 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:30:07.584109 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:30:37.583982 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:30:37.583942 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:31:07.583972 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:31:07.583930 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:31:37.583447 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:31:37.583408 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:32:07.583607 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:32:07.583560 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:32:37.583409 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:32:37.583329 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:33:07.583472 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:33:07.583432 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:33:37.583382 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:33:37.583338 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:34:07.583132 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:34:07.583091 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:34:37.583037 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:34:37.582998 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:35:07.583243 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:35:07.583202 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:35:37.584464 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:35:37.584376 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:36:07.583309 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:36:07.583269 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:36:37.586313 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:36:37.586269 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:36:50.303554 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.303508 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc"] Nov 03 00:36:50.304078 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.303865 5114 cpu_manager.go:414] "containerMap: removing container" podUID="2e380280-d7c8-4e7f-9b88-fe7c50271b2a" containerName="main" Nov 03 00:36:50.304078 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.303876 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e380280-d7c8-4e7f-9b88-fe7c50271b2a" containerName="main" Nov 03 00:36:50.304078 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.303904 5114 cpu_manager.go:414] "containerMap: removing container" podUID="a17b8a18-1102-48b2-98ec-5b5adf9819df" containerName="main" Nov 03 00:36:50.304078 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.303909 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a17b8a18-1102-48b2-98ec-5b5adf9819df" containerName="main" Nov 03 00:36:50.304078 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.303969 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="a17b8a18-1102-48b2-98ec-5b5adf9819df" containerName="main" Nov 03 00:36:50.304078 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.303978 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e380280-d7c8-4e7f-9b88-fe7c50271b2a" containerName="main" Nov 03 00:36:50.307049 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.307023 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:36:50.366774 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.366721 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n7m2\" (UniqueName: \"kubernetes.io/projected/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-kube-api-access-4n7m2\") pod \"simulator-single-main-649f61-kserve-f9c97c898-tmmfc\" (UID: \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\") " pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:36:50.366960 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.366782 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-tmp-dir\") pod \"simulator-single-main-649f61-kserve-f9c97c898-tmmfc\" (UID: \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\") " pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:36:50.366960 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.366890 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-model-cache\") pod \"simulator-single-main-649f61-kserve-f9c97c898-tmmfc\" (UID: \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\") " pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:36:50.366960 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.366950 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-tls-certs\") pod \"simulator-single-main-649f61-kserve-f9c97c898-tmmfc\" (UID: \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\") " pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:36:50.367104 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.367039 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-home\") pod \"simulator-single-main-649f61-kserve-f9c97c898-tmmfc\" (UID: \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\") " pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:36:50.367162 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.367146 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-dshm\") pod \"simulator-single-main-649f61-kserve-f9c97c898-tmmfc\" (UID: \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\") " pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:36:50.408602 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.408553 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-649f61\"/\"openshift-service-ca.crt\"" Nov 03 00:36:50.408602 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.408589 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-649f61\"/\"default-dockercfg-lbzdx\"" Nov 03 00:36:50.408602 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.408553 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-649f61\"/\"kube-root-ca.crt\"" Nov 03 00:36:50.408881 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.408734 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-649f61\"/\"simulator-single-main-649f61-kserve-self-signed-certs\"" Nov 03 00:36:50.409982 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.409960 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc"] Nov 03 00:36:50.467826 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.467776 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-tls-certs\") pod \"simulator-single-main-649f61-kserve-f9c97c898-tmmfc\" (UID: \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\") " pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:36:50.467826 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.467827 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-home\") pod \"simulator-single-main-649f61-kserve-f9c97c898-tmmfc\" (UID: \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\") " pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:36:50.468140 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.467896 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-dshm\") pod \"simulator-single-main-649f61-kserve-f9c97c898-tmmfc\" (UID: \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\") " pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:36:50.468140 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.467953 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n7m2\" (UniqueName: \"kubernetes.io/projected/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-kube-api-access-4n7m2\") pod \"simulator-single-main-649f61-kserve-f9c97c898-tmmfc\" (UID: \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\") " pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:36:50.468140 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.467983 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-tmp-dir\") pod \"simulator-single-main-649f61-kserve-f9c97c898-tmmfc\" (UID: \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\") " pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:36:50.468140 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.468022 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-model-cache\") pod \"simulator-single-main-649f61-kserve-f9c97c898-tmmfc\" (UID: \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\") " pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:36:50.468396 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.468369 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-home\") pod \"simulator-single-main-649f61-kserve-f9c97c898-tmmfc\" (UID: \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\") " pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:36:50.468565 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.468444 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-model-cache\") pod \"simulator-single-main-649f61-kserve-f9c97c898-tmmfc\" (UID: \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\") " pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:36:50.468701 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.468672 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-tmp-dir\") pod \"simulator-single-main-649f61-kserve-f9c97c898-tmmfc\" (UID: \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\") " pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:36:50.470977 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.470943 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-dshm\") pod \"simulator-single-main-649f61-kserve-f9c97c898-tmmfc\" (UID: \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\") " pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:36:50.471367 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.471346 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-tls-certs\") pod \"simulator-single-main-649f61-kserve-f9c97c898-tmmfc\" (UID: \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\") " pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:36:50.529399 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.529366 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n7m2\" (UniqueName: \"kubernetes.io/projected/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-kube-api-access-4n7m2\") pod \"simulator-single-main-649f61-kserve-f9c97c898-tmmfc\" (UID: \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\") " pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:36:50.618601 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:50.618469 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:36:51.000060 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:51.000028 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc"] Nov 03 00:36:51.002532 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:36:51.002494 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c8bc0e9_b648_476d_bd42_255d3d2b72fc.slice/crio-ed9af266467a365ed1065085da71ba242f621e149fe37d2159592e4eda9890be WatchSource:0}: Error finding container ed9af266467a365ed1065085da71ba242f621e149fe37d2159592e4eda9890be: Status 404 returned error can't find the container with id ed9af266467a365ed1065085da71ba242f621e149fe37d2159592e4eda9890be Nov 03 00:36:51.405146 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:51.405102 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" event={"ID":"9c8bc0e9-b648-476d-bd42-255d3d2b72fc","Type":"ContainerStarted","Data":"d1357373ee760c1a1c7b28ea866b0254d0276b07139e14baf01c6ae1134e323c"} Nov 03 00:36:51.405680 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:51.405163 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" event={"ID":"9c8bc0e9-b648-476d-bd42-255d3d2b72fc","Type":"ContainerStarted","Data":"ed9af266467a365ed1065085da71ba242f621e149fe37d2159592e4eda9890be"} Nov 03 00:36:51.521310 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:51.521255 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" podStartSLOduration=1.521240593 podStartE2EDuration="1.521240593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:51.521005322 +0000 UTC m=+6486.874038698" watchObservedRunningTime="2026-11-03 00:36:51.521240593 +0000 UTC m=+6486.874273969" Nov 03 00:36:51.619102 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:36:51.619040 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:37:00.619534 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:37:00.619468 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:37:00.632465 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:37:00.632428 5114 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:37:01.459656 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:37:01.459625 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:37:07.584521 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:37:07.584454 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:37:37.583393 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:37:37.583351 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:37:51.537869 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:37:51.537832 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc"] Nov 03 00:38:06.558463 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:06.558416 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" podUID="9c8bc0e9-b648-476d-bd42-255d3d2b72fc" containerName="main" containerID="cri-o://d1357373ee760c1a1c7b28ea866b0254d0276b07139e14baf01c6ae1134e323c" gracePeriod=45 Nov 03 00:38:06.732945 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:06.732894 5114 generic.go:356] "Generic (PLEG): container finished" podID="9c8bc0e9-b648-476d-bd42-255d3d2b72fc" containerID="d1357373ee760c1a1c7b28ea866b0254d0276b07139e14baf01c6ae1134e323c" exitCode=0 Nov 03 00:38:06.733119 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:06.732956 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" event={"ID":"9c8bc0e9-b648-476d-bd42-255d3d2b72fc","Type":"ContainerDied","Data":"d1357373ee760c1a1c7b28ea866b0254d0276b07139e14baf01c6ae1134e323c"} Nov 03 00:38:06.802901 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:06.802877 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:38:06.867306 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:06.867212 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-model-cache\") pod \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\" (UID: \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\") " Nov 03 00:38:06.867306 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:06.867302 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-tmp-dir\") pod \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\" (UID: \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\") " Nov 03 00:38:06.867605 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:06.867329 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-kube-api-access-4n7m2\" (UniqueName: \"kubernetes.io/projected/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-kube-api-access-4n7m2\") pod \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\" (UID: \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\") " Nov 03 00:38:06.867605 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:06.867364 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-tls-certs\") pod \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\" (UID: \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\") " Nov 03 00:38:06.867605 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:06.867382 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-home\" (UniqueName: \"kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-home\") pod \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\" (UID: \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\") " Nov 03 00:38:06.867605 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:06.867445 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-dshm\") pod \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\" (UID: \"9c8bc0e9-b648-476d-bd42-255d3d2b72fc\") " Nov 03 00:38:06.867605 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:06.867566 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-model-cache" pod "9c8bc0e9-b648-476d-bd42-255d3d2b72fc" (UID: "9c8bc0e9-b648-476d-bd42-255d3d2b72fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:38:06.867902 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:06.867651 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-tmp-dir" pod "9c8bc0e9-b648-476d-bd42-255d3d2b72fc" (UID: "9c8bc0e9-b648-476d-bd42-255d3d2b72fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:38:06.867902 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:06.867729 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-home" pod "9c8bc0e9-b648-476d-bd42-255d3d2b72fc" (UID: "9c8bc0e9-b648-476d-bd42-255d3d2b72fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:38:06.867902 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:06.867786 5114 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-model-cache\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:06.867902 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:06.867801 5114 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-tmp-dir\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:06.869940 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:06.869910 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-dshm" pod "9c8bc0e9-b648-476d-bd42-255d3d2b72fc" (UID: "9c8bc0e9-b648-476d-bd42-255d3d2b72fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:38:06.870052 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:06.869962 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-kube-api-access-4n7m2" pod "9c8bc0e9-b648-476d-bd42-255d3d2b72fc" (UID: "9c8bc0e9-b648-476d-bd42-255d3d2b72fc"). InnerVolumeSpecName "kube-api-access-4n7m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:38:06.870052 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:06.870024 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-tls-certs" pod "9c8bc0e9-b648-476d-bd42-255d3d2b72fc" (UID: "9c8bc0e9-b648-476d-bd42-255d3d2b72fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:38:06.969074 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:06.969034 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n7m2\" (UniqueName: \"kubernetes.io/projected/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-kube-api-access-4n7m2\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:06.969074 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:06.969071 5114 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-tls-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:06.969074 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:06.969082 5114 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-home\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:06.969369 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:06.969090 5114 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c8bc0e9-b648-476d-bd42-255d3d2b72fc-dshm\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:07.584386 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:38:07.584346 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:38:07.738474 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:07.738447 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" Nov 03 00:38:07.738711 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:07.738504 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc" event={"ID":"9c8bc0e9-b648-476d-bd42-255d3d2b72fc","Type":"ContainerDied","Data":"ed9af266467a365ed1065085da71ba242f621e149fe37d2159592e4eda9890be"} Nov 03 00:38:07.738711 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:07.738555 5114 scope.go:122] "RemoveContainer" containerID="d1357373ee760c1a1c7b28ea866b0254d0276b07139e14baf01c6ae1134e323c" Nov 03 00:38:07.902323 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:07.902213 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc"] Nov 03 00:38:07.951328 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:07.951289 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-649f61/simulator-single-main-649f61-kserve-f9c97c898-tmmfc"] Nov 03 00:38:08.792220 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:38:08.792189 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c8bc0e9-b648-476d-bd42-255d3d2b72fc" path="/var/lib/kubelet/pods/9c8bc0e9-b648-476d-bd42-255d3d2b72fc/volumes" Nov 03 00:38:37.582914 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:38:37.582827 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:39:07.583463 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:39:07.583422 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:39:37.583247 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:39:37.583203 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:40:07.583356 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:40:07.583315 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:40:37.583816 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:40:37.583771 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:41:07.599183 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:41:07.599144 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:41:37.583962 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:41:37.583874 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:41:42.954645 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:42.954604 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-94b5f474-v8lbx"] Nov 03 00:41:42.955289 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:42.954821 5114 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-94b5f474-v8lbx" podUID="78e093a3-d73a-476f-b2c9-a6f5c452c401" containerName="maas-api" containerID="cri-o://f22e7a66254ea04552d5788635b6df74ef1cc2d5a858f23dd4e274224eccf1ff" gracePeriod=30 Nov 03 00:41:43.578671 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:43.578622 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-94b5f474-v8lbx" podUID="78e093a3-d73a-476f-b2c9-a6f5c452c401" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.25:8443/health\": dial tcp 10.134.0.25:8443: connect: connection refused" Nov 03 00:41:44.212339 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:44.212310 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-94b5f474-v8lbx" Nov 03 00:41:44.337812 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:44.337771 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78e093a3-d73a-476f-b2c9-a6f5c452c401-kube-api-access-d5nnc\" (UniqueName: \"kubernetes.io/projected/78e093a3-d73a-476f-b2c9-a6f5c452c401-kube-api-access-d5nnc\") pod \"78e093a3-d73a-476f-b2c9-a6f5c452c401\" (UID: \"78e093a3-d73a-476f-b2c9-a6f5c452c401\") " Nov 03 00:41:44.338355 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:44.337841 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78e093a3-d73a-476f-b2c9-a6f5c452c401-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e093a3-d73a-476f-b2c9-a6f5c452c401-openshift-service-ca\") pod \"78e093a3-d73a-476f-b2c9-a6f5c452c401\" (UID: \"78e093a3-d73a-476f-b2c9-a6f5c452c401\") " Nov 03 00:41:44.338355 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:44.337954 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78e093a3-d73a-476f-b2c9-a6f5c452c401-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78e093a3-d73a-476f-b2c9-a6f5c452c401-maas-api-tls\") pod \"78e093a3-d73a-476f-b2c9-a6f5c452c401\" (UID: \"78e093a3-d73a-476f-b2c9-a6f5c452c401\") " Nov 03 00:41:44.338355 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:44.337995 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78e093a3-d73a-476f-b2c9-a6f5c452c401-metrics-certs\" (UniqueName: \"kubernetes.io/secret/78e093a3-d73a-476f-b2c9-a6f5c452c401-metrics-certs\") pod \"78e093a3-d73a-476f-b2c9-a6f5c452c401\" (UID: \"78e093a3-d73a-476f-b2c9-a6f5c452c401\") " Nov 03 00:41:44.338657 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:44.338366 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e093a3-d73a-476f-b2c9-a6f5c452c401-openshift-service-ca" pod "78e093a3-d73a-476f-b2c9-a6f5c452c401" (UID: "78e093a3-d73a-476f-b2c9-a6f5c452c401"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:41:44.340464 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:44.340432 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e093a3-d73a-476f-b2c9-a6f5c452c401-kube-api-access-d5nnc" pod "78e093a3-d73a-476f-b2c9-a6f5c452c401" (UID: "78e093a3-d73a-476f-b2c9-a6f5c452c401"). InnerVolumeSpecName "kube-api-access-d5nnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:41:44.340592 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:44.340524 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78e093a3-d73a-476f-b2c9-a6f5c452c401-metrics-certs" pod "78e093a3-d73a-476f-b2c9-a6f5c452c401" (UID: "78e093a3-d73a-476f-b2c9-a6f5c452c401"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:41:44.340691 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:44.340668 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78e093a3-d73a-476f-b2c9-a6f5c452c401-maas-api-tls" pod "78e093a3-d73a-476f-b2c9-a6f5c452c401" (UID: "78e093a3-d73a-476f-b2c9-a6f5c452c401"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:41:44.438924 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:44.438864 5114 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e093a3-d73a-476f-b2c9-a6f5c452c401-openshift-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:44.438924 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:44.438903 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78e093a3-d73a-476f-b2c9-a6f5c452c401-maas-api-tls\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:44.438924 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:44.438922 5114 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78e093a3-d73a-476f-b2c9-a6f5c452c401-metrics-certs\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:44.438924 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:44.438936 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5nnc\" (UniqueName: \"kubernetes.io/projected/78e093a3-d73a-476f-b2c9-a6f5c452c401-kube-api-access-d5nnc\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:44.754449 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:44.754355 5114 generic.go:356] "Generic (PLEG): container finished" podID="78e093a3-d73a-476f-b2c9-a6f5c452c401" containerID="f22e7a66254ea04552d5788635b6df74ef1cc2d5a858f23dd4e274224eccf1ff" exitCode=0 Nov 03 00:41:44.754449 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:44.754414 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-94b5f474-v8lbx" Nov 03 00:41:44.754737 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:44.754421 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-94b5f474-v8lbx" event={"ID":"78e093a3-d73a-476f-b2c9-a6f5c452c401","Type":"ContainerDied","Data":"f22e7a66254ea04552d5788635b6df74ef1cc2d5a858f23dd4e274224eccf1ff"} Nov 03 00:41:44.754737 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:44.754590 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-94b5f474-v8lbx" event={"ID":"78e093a3-d73a-476f-b2c9-a6f5c452c401","Type":"ContainerDied","Data":"54c2298a30e25b1f4ffb5841fa7cbb27ce53c3e23b50eb094cf6b754f6a881c2"} Nov 03 00:41:44.754737 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:44.754624 5114 scope.go:122] "RemoveContainer" containerID="f22e7a66254ea04552d5788635b6df74ef1cc2d5a858f23dd4e274224eccf1ff" Nov 03 00:41:44.768171 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:44.768145 5114 scope.go:122] "RemoveContainer" containerID="f22e7a66254ea04552d5788635b6df74ef1cc2d5a858f23dd4e274224eccf1ff" Nov 03 00:41:44.768647 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:41:44.768616 5114 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f22e7a66254ea04552d5788635b6df74ef1cc2d5a858f23dd4e274224eccf1ff\": container with ID starting with f22e7a66254ea04552d5788635b6df74ef1cc2d5a858f23dd4e274224eccf1ff not found: ID does not exist" containerID="f22e7a66254ea04552d5788635b6df74ef1cc2d5a858f23dd4e274224eccf1ff" Nov 03 00:41:44.768794 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:44.768658 5114 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f22e7a66254ea04552d5788635b6df74ef1cc2d5a858f23dd4e274224eccf1ff"} err="failed to get container status \"f22e7a66254ea04552d5788635b6df74ef1cc2d5a858f23dd4e274224eccf1ff\": rpc error: code = NotFound desc = could not find container \"f22e7a66254ea04552d5788635b6df74ef1cc2d5a858f23dd4e274224eccf1ff\": container with ID starting with f22e7a66254ea04552d5788635b6df74ef1cc2d5a858f23dd4e274224eccf1ff not found: ID does not exist" Nov 03 00:41:44.919359 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:44.919313 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-94b5f474-v8lbx"] Nov 03 00:41:44.968280 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:44.968239 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-94b5f474-v8lbx"] Nov 03 00:41:46.791745 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:46.791711 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78e093a3-d73a-476f-b2c9-a6f5c452c401" path="/var/lib/kubelet/pods/78e093a3-d73a-476f-b2c9-a6f5c452c401/volumes" Nov 03 00:41:51.329038 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:51.328991 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d056df3-xq7sd"] Nov 03 00:41:51.329464 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:51.329315 5114 cpu_manager.go:414] "containerMap: removing container" podUID="9c8bc0e9-b648-476d-bd42-255d3d2b72fc" containerName="main" Nov 03 00:41:51.329464 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:51.329325 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c8bc0e9-b648-476d-bd42-255d3d2b72fc" containerName="main" Nov 03 00:41:51.329464 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:51.329335 5114 cpu_manager.go:414] "containerMap: removing container" podUID="78e093a3-d73a-476f-b2c9-a6f5c452c401" containerName="maas-api" Nov 03 00:41:51.329464 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:51.329340 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e093a3-d73a-476f-b2c9-a6f5c452c401" containerName="maas-api" Nov 03 00:41:51.329464 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:51.329396 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c8bc0e9-b648-476d-bd42-255d3d2b72fc" containerName="main" Nov 03 00:41:51.329464 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:51.329406 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e093a3-d73a-476f-b2c9-a6f5c452c401" containerName="maas-api" Nov 03 00:41:51.333774 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:51.333756 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d056df3-xq7sd" Nov 03 00:41:51.388652 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:51.388615 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:41:51.390182 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:51.390143 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d056df3-xq7sd"] Nov 03 00:41:51.427005 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:51.426972 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hmvk\" (UniqueName: \"kubernetes.io/projected/754056bc-5d48-4e08-b6f0-e967ad2787ae-kube-api-access-2hmvk\") pod \"maas-api-revoke-sub-7d056df3-xq7sd\" (UID: \"754056bc-5d48-4e08-b6f0-e967ad2787ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d056df3-xq7sd" Nov 03 00:41:51.427207 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:51.427030 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/754056bc-5d48-4e08-b6f0-e967ad2787ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d056df3-xq7sd\" (UID: \"754056bc-5d48-4e08-b6f0-e967ad2787ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d056df3-xq7sd" Nov 03 00:41:51.427207 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:51.427185 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/754056bc-5d48-4e08-b6f0-e967ad2787ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d056df3-xq7sd\" (UID: \"754056bc-5d48-4e08-b6f0-e967ad2787ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d056df3-xq7sd" Nov 03 00:41:51.437931 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:51.437887 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:41:51.437931 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:51.437919 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:41:51.438153 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:51.437960 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:41:51.527867 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:51.527831 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/754056bc-5d48-4e08-b6f0-e967ad2787ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d056df3-xq7sd\" (UID: \"754056bc-5d48-4e08-b6f0-e967ad2787ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d056df3-xq7sd" Nov 03 00:41:51.528057 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:51.527900 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/754056bc-5d48-4e08-b6f0-e967ad2787ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d056df3-xq7sd\" (UID: \"754056bc-5d48-4e08-b6f0-e967ad2787ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d056df3-xq7sd" Nov 03 00:41:51.528057 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:51.527963 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hmvk\" (UniqueName: \"kubernetes.io/projected/754056bc-5d48-4e08-b6f0-e967ad2787ae-kube-api-access-2hmvk\") pod \"maas-api-revoke-sub-7d056df3-xq7sd\" (UID: \"754056bc-5d48-4e08-b6f0-e967ad2787ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d056df3-xq7sd" Nov 03 00:41:51.528648 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:51.528619 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/754056bc-5d48-4e08-b6f0-e967ad2787ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d056df3-xq7sd\" (UID: \"754056bc-5d48-4e08-b6f0-e967ad2787ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d056df3-xq7sd" Nov 03 00:41:51.530531 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:51.530512 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/754056bc-5d48-4e08-b6f0-e967ad2787ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d056df3-xq7sd\" (UID: \"754056bc-5d48-4e08-b6f0-e967ad2787ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d056df3-xq7sd" Nov 03 00:41:51.588943 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:51.588853 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hmvk\" (UniqueName: \"kubernetes.io/projected/754056bc-5d48-4e08-b6f0-e967ad2787ae-kube-api-access-2hmvk\") pod \"maas-api-revoke-sub-7d056df3-xq7sd\" (UID: \"754056bc-5d48-4e08-b6f0-e967ad2787ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d056df3-xq7sd" Nov 03 00:41:51.646831 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:51.646771 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d056df3-xq7sd" Nov 03 00:41:52.021997 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:52.021899 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d056df3-xq7sd"] Nov 03 00:41:52.025090 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:41:52.025044 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod754056bc_5d48_4e08_b6f0_e967ad2787ae.slice/crio-022095c6ed711e0d757995e89af44a93cf38ec2ce8dc3dd801cbcbecf0d98d27 WatchSource:0}: Error finding container 022095c6ed711e0d757995e89af44a93cf38ec2ce8dc3dd801cbcbecf0d98d27: Status 404 returned error can't find the container with id 022095c6ed711e0d757995e89af44a93cf38ec2ce8dc3dd801cbcbecf0d98d27 Nov 03 00:41:52.809470 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:52.809432 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d056df3-xq7sd" event={"ID":"754056bc-5d48-4e08-b6f0-e967ad2787ae","Type":"ContainerStarted","Data":"c24e39025374b0a7a6ec230f33e9b66e47f49178ab1159510082828ef0c0a00e"} Nov 03 00:41:52.809960 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:52.809510 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d056df3-xq7sd" event={"ID":"754056bc-5d48-4e08-b6f0-e967ad2787ae","Type":"ContainerStarted","Data":"022095c6ed711e0d757995e89af44a93cf38ec2ce8dc3dd801cbcbecf0d98d27"} Nov 03 00:41:52.925316 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:52.925261 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d056df3-xq7sd" podStartSLOduration=1.925246933 podStartE2EDuration="1.925246933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:52.924362773 +0000 UTC m=+6788.277396148" watchObservedRunningTime="2026-11-03 00:41:52.925246933 +0000 UTC m=+6788.278280308" Nov 03 00:41:53.815324 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:53.815279 5114 generic.go:356] "Generic (PLEG): container finished" podID="754056bc-5d48-4e08-b6f0-e967ad2787ae" containerID="c24e39025374b0a7a6ec230f33e9b66e47f49178ab1159510082828ef0c0a00e" exitCode=0 Nov 03 00:41:53.815782 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:53.815368 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d056df3-xq7sd" event={"ID":"754056bc-5d48-4e08-b6f0-e967ad2787ae","Type":"ContainerDied","Data":"c24e39025374b0a7a6ec230f33e9b66e47f49178ab1159510082828ef0c0a00e"} Nov 03 00:41:54.952984 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:54.952953 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d056df3-xq7sd" Nov 03 00:41:55.063301 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:55.063266 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/754056bc-5d48-4e08-b6f0-e967ad2787ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/754056bc-5d48-4e08-b6f0-e967ad2787ae-maas-api-xks-ca\") pod \"754056bc-5d48-4e08-b6f0-e967ad2787ae\" (UID: \"754056bc-5d48-4e08-b6f0-e967ad2787ae\") " Nov 03 00:41:55.063523 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:55.063344 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/754056bc-5d48-4e08-b6f0-e967ad2787ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/754056bc-5d48-4e08-b6f0-e967ad2787ae-maas-api-openshift-ca\") pod \"754056bc-5d48-4e08-b6f0-e967ad2787ae\" (UID: \"754056bc-5d48-4e08-b6f0-e967ad2787ae\") " Nov 03 00:41:55.063523 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:55.063412 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/754056bc-5d48-4e08-b6f0-e967ad2787ae-kube-api-access-2hmvk\" (UniqueName: \"kubernetes.io/projected/754056bc-5d48-4e08-b6f0-e967ad2787ae-kube-api-access-2hmvk\") pod \"754056bc-5d48-4e08-b6f0-e967ad2787ae\" (UID: \"754056bc-5d48-4e08-b6f0-e967ad2787ae\") " Nov 03 00:41:55.063800 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:55.063774 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/754056bc-5d48-4e08-b6f0-e967ad2787ae-maas-api-openshift-ca" pod "754056bc-5d48-4e08-b6f0-e967ad2787ae" (UID: "754056bc-5d48-4e08-b6f0-e967ad2787ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:41:55.065829 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:55.065802 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/754056bc-5d48-4e08-b6f0-e967ad2787ae-maas-api-xks-ca" pod "754056bc-5d48-4e08-b6f0-e967ad2787ae" (UID: "754056bc-5d48-4e08-b6f0-e967ad2787ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:41:55.065974 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:55.065860 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/754056bc-5d48-4e08-b6f0-e967ad2787ae-kube-api-access-2hmvk" pod "754056bc-5d48-4e08-b6f0-e967ad2787ae" (UID: "754056bc-5d48-4e08-b6f0-e967ad2787ae"). InnerVolumeSpecName "kube-api-access-2hmvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:41:55.164263 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:55.164224 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hmvk\" (UniqueName: \"kubernetes.io/projected/754056bc-5d48-4e08-b6f0-e967ad2787ae-kube-api-access-2hmvk\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:55.164263 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:55.164256 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/754056bc-5d48-4e08-b6f0-e967ad2787ae-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:55.164263 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:55.164266 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/754056bc-5d48-4e08-b6f0-e967ad2787ae-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:55.831432 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:55.831399 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d056df3-xq7sd" Nov 03 00:41:55.831669 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:55.831432 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d056df3-xq7sd" event={"ID":"754056bc-5d48-4e08-b6f0-e967ad2787ae","Type":"ContainerDied","Data":"022095c6ed711e0d757995e89af44a93cf38ec2ce8dc3dd801cbcbecf0d98d27"} Nov 03 00:41:55.831669 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:41:55.831546 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="022095c6ed711e0d757995e89af44a93cf38ec2ce8dc3dd801cbcbecf0d98d27" Nov 03 00:42:02.082374 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:02.082334 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd3759ee-rs6zs"] Nov 03 00:42:02.082819 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:02.082680 5114 cpu_manager.go:414] "containerMap: removing container" podUID="754056bc-5d48-4e08-b6f0-e967ad2787ae" containerName="revoke-keys" Nov 03 00:42:02.082819 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:02.082692 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="754056bc-5d48-4e08-b6f0-e967ad2787ae" containerName="revoke-keys" Nov 03 00:42:02.082819 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:02.082769 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="754056bc-5d48-4e08-b6f0-e967ad2787ae" containerName="revoke-keys" Nov 03 00:42:02.085856 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:02.085834 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3759ee-rs6zs" Nov 03 00:42:02.138190 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:02.138147 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:42:02.187452 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:02.187413 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:42:02.187670 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:02.187469 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:42:02.187670 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:02.187529 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:42:02.188732 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:02.188710 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd3759ee-rs6zs"] Nov 03 00:42:02.223428 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:02.223389 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4da03fdb-612b-4dc9-9244-439b9073a10f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd3759ee-rs6zs\" (UID: \"4da03fdb-612b-4dc9-9244-439b9073a10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3759ee-rs6zs" Nov 03 00:42:02.223428 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:02.223428 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4da03fdb-612b-4dc9-9244-439b9073a10f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd3759ee-rs6zs\" (UID: \"4da03fdb-612b-4dc9-9244-439b9073a10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3759ee-rs6zs" Nov 03 00:42:02.223705 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:02.223640 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn8p4\" (UniqueName: \"kubernetes.io/projected/4da03fdb-612b-4dc9-9244-439b9073a10f-kube-api-access-nn8p4\") pod \"maas-api-revoke-sub-bd3759ee-rs6zs\" (UID: \"4da03fdb-612b-4dc9-9244-439b9073a10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3759ee-rs6zs" Nov 03 00:42:02.324428 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:02.324384 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn8p4\" (UniqueName: \"kubernetes.io/projected/4da03fdb-612b-4dc9-9244-439b9073a10f-kube-api-access-nn8p4\") pod \"maas-api-revoke-sub-bd3759ee-rs6zs\" (UID: \"4da03fdb-612b-4dc9-9244-439b9073a10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3759ee-rs6zs" Nov 03 00:42:02.324679 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:02.324477 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4da03fdb-612b-4dc9-9244-439b9073a10f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd3759ee-rs6zs\" (UID: \"4da03fdb-612b-4dc9-9244-439b9073a10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3759ee-rs6zs" Nov 03 00:42:02.324679 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:02.324531 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4da03fdb-612b-4dc9-9244-439b9073a10f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd3759ee-rs6zs\" (UID: \"4da03fdb-612b-4dc9-9244-439b9073a10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3759ee-rs6zs" Nov 03 00:42:02.325274 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:02.325251 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4da03fdb-612b-4dc9-9244-439b9073a10f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd3759ee-rs6zs\" (UID: \"4da03fdb-612b-4dc9-9244-439b9073a10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3759ee-rs6zs" Nov 03 00:42:02.327167 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:02.327144 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4da03fdb-612b-4dc9-9244-439b9073a10f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd3759ee-rs6zs\" (UID: \"4da03fdb-612b-4dc9-9244-439b9073a10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3759ee-rs6zs" Nov 03 00:42:02.385887 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:02.385806 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn8p4\" (UniqueName: \"kubernetes.io/projected/4da03fdb-612b-4dc9-9244-439b9073a10f-kube-api-access-nn8p4\") pod \"maas-api-revoke-sub-bd3759ee-rs6zs\" (UID: \"4da03fdb-612b-4dc9-9244-439b9073a10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3759ee-rs6zs" Nov 03 00:42:02.399515 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:02.399467 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3759ee-rs6zs" Nov 03 00:42:02.772075 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:02.772043 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd3759ee-rs6zs"] Nov 03 00:42:02.774770 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:42:02.774737 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4da03fdb_612b_4dc9_9244_439b9073a10f.slice/crio-43d87452cdcd4eca311a94b0bb742a9a920f6aa5cc5f788549d3827423b5aee1 WatchSource:0}: Error finding container 43d87452cdcd4eca311a94b0bb742a9a920f6aa5cc5f788549d3827423b5aee1: Status 404 returned error can't find the container with id 43d87452cdcd4eca311a94b0bb742a9a920f6aa5cc5f788549d3827423b5aee1 Nov 03 00:42:02.861273 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:02.861234 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3759ee-rs6zs" event={"ID":"4da03fdb-612b-4dc9-9244-439b9073a10f","Type":"ContainerStarted","Data":"43d87452cdcd4eca311a94b0bb742a9a920f6aa5cc5f788549d3827423b5aee1"} Nov 03 00:42:03.871808 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:03.871775 5114 generic.go:356] "Generic (PLEG): container finished" podID="4da03fdb-612b-4dc9-9244-439b9073a10f" containerID="a9be850c3e5036bef3671cbce8417f8d9846818bd11d78308c2e86a2a91cfc15" exitCode=0 Nov 03 00:42:03.872270 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:03.871863 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3759ee-rs6zs" event={"ID":"4da03fdb-612b-4dc9-9244-439b9073a10f","Type":"ContainerDied","Data":"a9be850c3e5036bef3671cbce8417f8d9846818bd11d78308c2e86a2a91cfc15"} Nov 03 00:42:05.017552 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:05.017520 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3759ee-rs6zs" Nov 03 00:42:05.150059 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:05.149967 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4da03fdb-612b-4dc9-9244-439b9073a10f-kube-api-access-nn8p4\" (UniqueName: \"kubernetes.io/projected/4da03fdb-612b-4dc9-9244-439b9073a10f-kube-api-access-nn8p4\") pod \"4da03fdb-612b-4dc9-9244-439b9073a10f\" (UID: \"4da03fdb-612b-4dc9-9244-439b9073a10f\") " Nov 03 00:42:05.150059 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:05.150021 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4da03fdb-612b-4dc9-9244-439b9073a10f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4da03fdb-612b-4dc9-9244-439b9073a10f-maas-api-openshift-ca\") pod \"4da03fdb-612b-4dc9-9244-439b9073a10f\" (UID: \"4da03fdb-612b-4dc9-9244-439b9073a10f\") " Nov 03 00:42:05.150312 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:05.150137 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4da03fdb-612b-4dc9-9244-439b9073a10f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4da03fdb-612b-4dc9-9244-439b9073a10f-maas-api-xks-ca\") pod \"4da03fdb-612b-4dc9-9244-439b9073a10f\" (UID: \"4da03fdb-612b-4dc9-9244-439b9073a10f\") " Nov 03 00:42:05.150516 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:05.150450 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4da03fdb-612b-4dc9-9244-439b9073a10f-maas-api-openshift-ca" pod "4da03fdb-612b-4dc9-9244-439b9073a10f" (UID: "4da03fdb-612b-4dc9-9244-439b9073a10f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:05.152410 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:05.152380 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4da03fdb-612b-4dc9-9244-439b9073a10f-maas-api-xks-ca" pod "4da03fdb-612b-4dc9-9244-439b9073a10f" (UID: "4da03fdb-612b-4dc9-9244-439b9073a10f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:05.152567 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:05.152436 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4da03fdb-612b-4dc9-9244-439b9073a10f-kube-api-access-nn8p4" pod "4da03fdb-612b-4dc9-9244-439b9073a10f" (UID: "4da03fdb-612b-4dc9-9244-439b9073a10f"). InnerVolumeSpecName "kube-api-access-nn8p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:05.251406 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:05.251363 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn8p4\" (UniqueName: \"kubernetes.io/projected/4da03fdb-612b-4dc9-9244-439b9073a10f-kube-api-access-nn8p4\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:05.251406 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:05.251396 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4da03fdb-612b-4dc9-9244-439b9073a10f-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:05.251406 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:05.251406 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4da03fdb-612b-4dc9-9244-439b9073a10f-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:05.882251 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:05.882216 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3759ee-rs6zs" Nov 03 00:42:05.882449 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:05.882259 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3759ee-rs6zs" event={"ID":"4da03fdb-612b-4dc9-9244-439b9073a10f","Type":"ContainerDied","Data":"43d87452cdcd4eca311a94b0bb742a9a920f6aa5cc5f788549d3827423b5aee1"} Nov 03 00:42:05.882449 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:05.882304 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43d87452cdcd4eca311a94b0bb742a9a920f6aa5cc5f788549d3827423b5aee1" Nov 03 00:42:07.582736 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:42:07.582693 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:42:23.702731 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:23.702676 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7zbtz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:42:23.703310 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:23.702756 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7zbtz" podUID="19ba6ae1-0768-4641-9026-bce647422531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:42:23.919829 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:23.919780 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-92jjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:42:23.920196 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:23.920140 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-92jjh" podUID="b4e963bb-8c2f-4628-a009-ebbebd62394f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:42:37.583681 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:42:37.583639 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:42:37.670188 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:37.670146 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f1b7d9f-rbxjm"] Nov 03 00:42:37.670526 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:37.670512 5114 cpu_manager.go:414] "containerMap: removing container" podUID="4da03fdb-612b-4dc9-9244-439b9073a10f" containerName="revoke-keys" Nov 03 00:42:37.670591 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:37.670528 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4da03fdb-612b-4dc9-9244-439b9073a10f" containerName="revoke-keys" Nov 03 00:42:37.670640 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:37.670603 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="4da03fdb-612b-4dc9-9244-439b9073a10f" containerName="revoke-keys" Nov 03 00:42:37.676315 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:37.676290 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1b7d9f-rbxjm" Nov 03 00:42:37.732602 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:37.732560 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:42:37.741774 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:37.741745 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slzd2\" (UniqueName: \"kubernetes.io/projected/603ad53d-b227-4870-95da-99ef1a795914-kube-api-access-slzd2\") pod \"maas-api-revoke-sub-5f1b7d9f-rbxjm\" (UID: \"603ad53d-b227-4870-95da-99ef1a795914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1b7d9f-rbxjm" Nov 03 00:42:37.741955 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:37.741782 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/603ad53d-b227-4870-95da-99ef1a795914-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f1b7d9f-rbxjm\" (UID: \"603ad53d-b227-4870-95da-99ef1a795914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1b7d9f-rbxjm" Nov 03 00:42:37.741955 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:37.741896 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/603ad53d-b227-4870-95da-99ef1a795914-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f1b7d9f-rbxjm\" (UID: \"603ad53d-b227-4870-95da-99ef1a795914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1b7d9f-rbxjm" Nov 03 00:42:37.780690 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:37.780651 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:42:37.780690 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:37.780655 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:42:37.780989 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:37.780719 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:42:37.782228 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:37.782199 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f1b7d9f-rbxjm"] Nov 03 00:42:37.842566 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:37.842452 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slzd2\" (UniqueName: \"kubernetes.io/projected/603ad53d-b227-4870-95da-99ef1a795914-kube-api-access-slzd2\") pod \"maas-api-revoke-sub-5f1b7d9f-rbxjm\" (UID: \"603ad53d-b227-4870-95da-99ef1a795914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1b7d9f-rbxjm" Nov 03 00:42:37.842566 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:37.842515 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/603ad53d-b227-4870-95da-99ef1a795914-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f1b7d9f-rbxjm\" (UID: \"603ad53d-b227-4870-95da-99ef1a795914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1b7d9f-rbxjm" Nov 03 00:42:37.842771 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:37.842576 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/603ad53d-b227-4870-95da-99ef1a795914-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f1b7d9f-rbxjm\" (UID: \"603ad53d-b227-4870-95da-99ef1a795914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1b7d9f-rbxjm" Nov 03 00:42:37.843207 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:37.843182 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/603ad53d-b227-4870-95da-99ef1a795914-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f1b7d9f-rbxjm\" (UID: \"603ad53d-b227-4870-95da-99ef1a795914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1b7d9f-rbxjm" Nov 03 00:42:37.845159 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:37.845125 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/603ad53d-b227-4870-95da-99ef1a795914-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f1b7d9f-rbxjm\" (UID: \"603ad53d-b227-4870-95da-99ef1a795914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1b7d9f-rbxjm" Nov 03 00:42:37.903443 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:37.903409 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slzd2\" (UniqueName: \"kubernetes.io/projected/603ad53d-b227-4870-95da-99ef1a795914-kube-api-access-slzd2\") pod \"maas-api-revoke-sub-5f1b7d9f-rbxjm\" (UID: \"603ad53d-b227-4870-95da-99ef1a795914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1b7d9f-rbxjm" Nov 03 00:42:37.989673 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:37.989628 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1b7d9f-rbxjm" Nov 03 00:42:38.364408 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:38.364373 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f1b7d9f-rbxjm"] Nov 03 00:42:38.367511 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:42:38.367466 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603ad53d_b227_4870_95da_99ef1a795914.slice/crio-76acb199025d6ba29b5e033780b7d91b5e1ba1b171c7886c25100560ae8bf862 WatchSource:0}: Error finding container 76acb199025d6ba29b5e033780b7d91b5e1ba1b171c7886c25100560ae8bf862: Status 404 returned error can't find the container with id 76acb199025d6ba29b5e033780b7d91b5e1ba1b171c7886c25100560ae8bf862 Nov 03 00:42:39.046689 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:39.046650 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1b7d9f-rbxjm" event={"ID":"603ad53d-b227-4870-95da-99ef1a795914","Type":"ContainerStarted","Data":"19106c0982e79c3adb3c0a9ff051c61b90f99f177a33bc1394586494b861bef6"} Nov 03 00:42:39.047114 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:39.046709 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1b7d9f-rbxjm" event={"ID":"603ad53d-b227-4870-95da-99ef1a795914","Type":"ContainerStarted","Data":"76acb199025d6ba29b5e033780b7d91b5e1ba1b171c7886c25100560ae8bf862"} Nov 03 00:42:39.163864 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:39.163787 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1b7d9f-rbxjm" podStartSLOduration=2.163767066 podStartE2EDuration="2.163767066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:42:39.162593536 +0000 UTC m=+6834.515626914" watchObservedRunningTime="2026-11-03 00:42:39.163767066 +0000 UTC m=+6834.516800443" Nov 03 00:42:40.052535 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:40.052477 5114 generic.go:356] "Generic (PLEG): container finished" podID="603ad53d-b227-4870-95da-99ef1a795914" containerID="19106c0982e79c3adb3c0a9ff051c61b90f99f177a33bc1394586494b861bef6" exitCode=0 Nov 03 00:42:40.052991 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:40.052587 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1b7d9f-rbxjm" event={"ID":"603ad53d-b227-4870-95da-99ef1a795914","Type":"ContainerDied","Data":"19106c0982e79c3adb3c0a9ff051c61b90f99f177a33bc1394586494b861bef6"} Nov 03 00:42:41.190531 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:41.190499 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1b7d9f-rbxjm" Nov 03 00:42:41.278302 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:41.278265 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/603ad53d-b227-4870-95da-99ef1a795914-kube-api-access-slzd2\" (UniqueName: \"kubernetes.io/projected/603ad53d-b227-4870-95da-99ef1a795914-kube-api-access-slzd2\") pod \"603ad53d-b227-4870-95da-99ef1a795914\" (UID: \"603ad53d-b227-4870-95da-99ef1a795914\") " Nov 03 00:42:41.278508 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:41.278377 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/603ad53d-b227-4870-95da-99ef1a795914-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/603ad53d-b227-4870-95da-99ef1a795914-maas-api-openshift-ca\") pod \"603ad53d-b227-4870-95da-99ef1a795914\" (UID: \"603ad53d-b227-4870-95da-99ef1a795914\") " Nov 03 00:42:41.278508 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:41.278413 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/603ad53d-b227-4870-95da-99ef1a795914-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/603ad53d-b227-4870-95da-99ef1a795914-maas-api-xks-ca\") pod \"603ad53d-b227-4870-95da-99ef1a795914\" (UID: \"603ad53d-b227-4870-95da-99ef1a795914\") " Nov 03 00:42:41.278777 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:41.278747 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603ad53d-b227-4870-95da-99ef1a795914-maas-api-openshift-ca" pod "603ad53d-b227-4870-95da-99ef1a795914" (UID: "603ad53d-b227-4870-95da-99ef1a795914"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:41.280720 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:41.280698 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603ad53d-b227-4870-95da-99ef1a795914-kube-api-access-slzd2" pod "603ad53d-b227-4870-95da-99ef1a795914" (UID: "603ad53d-b227-4870-95da-99ef1a795914"). InnerVolumeSpecName "kube-api-access-slzd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:41.280810 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:41.280705 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/603ad53d-b227-4870-95da-99ef1a795914-maas-api-xks-ca" pod "603ad53d-b227-4870-95da-99ef1a795914" (UID: "603ad53d-b227-4870-95da-99ef1a795914"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:41.379587 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:41.379467 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/603ad53d-b227-4870-95da-99ef1a795914-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:41.379587 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:41.379525 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/603ad53d-b227-4870-95da-99ef1a795914-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:41.379587 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:41.379538 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slzd2\" (UniqueName: \"kubernetes.io/projected/603ad53d-b227-4870-95da-99ef1a795914-kube-api-access-slzd2\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:42.063336 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:42.063305 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1b7d9f-rbxjm" Nov 03 00:42:42.063336 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:42.063310 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f1b7d9f-rbxjm" event={"ID":"603ad53d-b227-4870-95da-99ef1a795914","Type":"ContainerDied","Data":"76acb199025d6ba29b5e033780b7d91b5e1ba1b171c7886c25100560ae8bf862"} Nov 03 00:42:42.063596 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:42:42.063363 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76acb199025d6ba29b5e033780b7d91b5e1ba1b171c7886c25100560ae8bf862" Nov 03 00:43:07.583418 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:43:07.583384 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:43:22.393029 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:22.392980 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-443a2b88-qfkfq"] Nov 03 00:43:22.393453 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:22.393330 5114 cpu_manager.go:414] "containerMap: removing container" podUID="603ad53d-b227-4870-95da-99ef1a795914" containerName="revoke-keys" Nov 03 00:43:22.393453 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:22.393342 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="603ad53d-b227-4870-95da-99ef1a795914" containerName="revoke-keys" Nov 03 00:43:22.393453 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:22.393413 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="603ad53d-b227-4870-95da-99ef1a795914" containerName="revoke-keys" Nov 03 00:43:22.396703 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:22.396676 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a2b88-qfkfq" Nov 03 00:43:22.448692 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:22.448648 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:43:22.448692 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:22.448660 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:43:22.498213 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:22.498178 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:43:22.498213 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:22.498198 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:43:22.499736 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:22.499711 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-443a2b88-qfkfq"] Nov 03 00:43:22.513558 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:22.513520 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7961012-23c1-458a-83fe-0354645dc03a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-443a2b88-qfkfq\" (UID: \"e7961012-23c1-458a-83fe-0354645dc03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a2b88-qfkfq" Nov 03 00:43:22.513755 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:22.513568 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nkq4\" (UniqueName: \"kubernetes.io/projected/e7961012-23c1-458a-83fe-0354645dc03a-kube-api-access-5nkq4\") pod \"maas-api-revoke-sub-443a2b88-qfkfq\" (UID: \"e7961012-23c1-458a-83fe-0354645dc03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a2b88-qfkfq" Nov 03 00:43:22.513755 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:22.513697 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7961012-23c1-458a-83fe-0354645dc03a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-443a2b88-qfkfq\" (UID: \"e7961012-23c1-458a-83fe-0354645dc03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a2b88-qfkfq" Nov 03 00:43:22.614532 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:22.614462 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nkq4\" (UniqueName: \"kubernetes.io/projected/e7961012-23c1-458a-83fe-0354645dc03a-kube-api-access-5nkq4\") pod \"maas-api-revoke-sub-443a2b88-qfkfq\" (UID: \"e7961012-23c1-458a-83fe-0354645dc03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a2b88-qfkfq" Nov 03 00:43:22.614710 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:22.614585 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7961012-23c1-458a-83fe-0354645dc03a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-443a2b88-qfkfq\" (UID: \"e7961012-23c1-458a-83fe-0354645dc03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a2b88-qfkfq" Nov 03 00:43:22.614710 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:22.614623 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7961012-23c1-458a-83fe-0354645dc03a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-443a2b88-qfkfq\" (UID: \"e7961012-23c1-458a-83fe-0354645dc03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a2b88-qfkfq" Nov 03 00:43:22.615381 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:22.615359 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7961012-23c1-458a-83fe-0354645dc03a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-443a2b88-qfkfq\" (UID: \"e7961012-23c1-458a-83fe-0354645dc03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a2b88-qfkfq" Nov 03 00:43:22.617302 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:22.617279 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7961012-23c1-458a-83fe-0354645dc03a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-443a2b88-qfkfq\" (UID: \"e7961012-23c1-458a-83fe-0354645dc03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a2b88-qfkfq" Nov 03 00:43:22.675471 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:22.675395 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nkq4\" (UniqueName: \"kubernetes.io/projected/e7961012-23c1-458a-83fe-0354645dc03a-kube-api-access-5nkq4\") pod \"maas-api-revoke-sub-443a2b88-qfkfq\" (UID: \"e7961012-23c1-458a-83fe-0354645dc03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a2b88-qfkfq" Nov 03 00:43:22.710635 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:22.710588 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a2b88-qfkfq" Nov 03 00:43:23.082431 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:23.082255 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-443a2b88-qfkfq"] Nov 03 00:43:23.084934 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:43:23.084903 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7961012_23c1_458a_83fe_0354645dc03a.slice/crio-bb0dbee7e835ebbf56eb2e77f39f58604be145b97c0f7d3bf32a5f8f2b013d76 WatchSource:0}: Error finding container bb0dbee7e835ebbf56eb2e77f39f58604be145b97c0f7d3bf32a5f8f2b013d76: Status 404 returned error can't find the container with id bb0dbee7e835ebbf56eb2e77f39f58604be145b97c0f7d3bf32a5f8f2b013d76 Nov 03 00:43:23.265198 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:23.265160 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a2b88-qfkfq" event={"ID":"e7961012-23c1-458a-83fe-0354645dc03a","Type":"ContainerStarted","Data":"bb0dbee7e835ebbf56eb2e77f39f58604be145b97c0f7d3bf32a5f8f2b013d76"} Nov 03 00:43:24.271239 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:24.271140 5114 generic.go:356] "Generic (PLEG): container finished" podID="e7961012-23c1-458a-83fe-0354645dc03a" containerID="1a422115b83ead62abfd9249c0c0c2cf8955f99188fd989749f5c20f9bff3625" exitCode=0 Nov 03 00:43:24.271731 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:24.271228 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a2b88-qfkfq" event={"ID":"e7961012-23c1-458a-83fe-0354645dc03a","Type":"ContainerDied","Data":"1a422115b83ead62abfd9249c0c0c2cf8955f99188fd989749f5c20f9bff3625"} Nov 03 00:43:25.416021 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:25.415995 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a2b88-qfkfq" Nov 03 00:43:25.541177 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:25.541143 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7961012-23c1-458a-83fe-0354645dc03a-kube-api-access-5nkq4\" (UniqueName: \"kubernetes.io/projected/e7961012-23c1-458a-83fe-0354645dc03a-kube-api-access-5nkq4\") pod \"e7961012-23c1-458a-83fe-0354645dc03a\" (UID: \"e7961012-23c1-458a-83fe-0354645dc03a\") " Nov 03 00:43:25.541177 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:25.541180 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7961012-23c1-458a-83fe-0354645dc03a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7961012-23c1-458a-83fe-0354645dc03a-maas-api-openshift-ca\") pod \"e7961012-23c1-458a-83fe-0354645dc03a\" (UID: \"e7961012-23c1-458a-83fe-0354645dc03a\") " Nov 03 00:43:25.541475 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:25.541246 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7961012-23c1-458a-83fe-0354645dc03a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7961012-23c1-458a-83fe-0354645dc03a-maas-api-xks-ca\") pod \"e7961012-23c1-458a-83fe-0354645dc03a\" (UID: \"e7961012-23c1-458a-83fe-0354645dc03a\") " Nov 03 00:43:25.541621 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:25.541594 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7961012-23c1-458a-83fe-0354645dc03a-maas-api-openshift-ca" pod "e7961012-23c1-458a-83fe-0354645dc03a" (UID: "e7961012-23c1-458a-83fe-0354645dc03a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:43:25.543703 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:25.543668 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7961012-23c1-458a-83fe-0354645dc03a-maas-api-xks-ca" pod "e7961012-23c1-458a-83fe-0354645dc03a" (UID: "e7961012-23c1-458a-83fe-0354645dc03a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:43:25.543831 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:25.543701 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7961012-23c1-458a-83fe-0354645dc03a-kube-api-access-5nkq4" pod "e7961012-23c1-458a-83fe-0354645dc03a" (UID: "e7961012-23c1-458a-83fe-0354645dc03a"). InnerVolumeSpecName "kube-api-access-5nkq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:43:25.642244 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:25.642192 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nkq4\" (UniqueName: \"kubernetes.io/projected/e7961012-23c1-458a-83fe-0354645dc03a-kube-api-access-5nkq4\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:25.642244 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:25.642235 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7961012-23c1-458a-83fe-0354645dc03a-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:25.642244 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:25.642249 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7961012-23c1-458a-83fe-0354645dc03a-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:26.280668 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:26.280636 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a2b88-qfkfq" Nov 03 00:43:26.280863 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:26.280636 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a2b88-qfkfq" event={"ID":"e7961012-23c1-458a-83fe-0354645dc03a","Type":"ContainerDied","Data":"bb0dbee7e835ebbf56eb2e77f39f58604be145b97c0f7d3bf32a5f8f2b013d76"} Nov 03 00:43:26.280863 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:26.280759 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb0dbee7e835ebbf56eb2e77f39f58604be145b97c0f7d3bf32a5f8f2b013d76" Nov 03 00:43:33.146066 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:33.146020 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7de5d31-zhthq"] Nov 03 00:43:33.146638 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:33.146348 5114 cpu_manager.go:414] "containerMap: removing container" podUID="e7961012-23c1-458a-83fe-0354645dc03a" containerName="revoke-keys" Nov 03 00:43:33.146638 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:33.146359 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7961012-23c1-458a-83fe-0354645dc03a" containerName="revoke-keys" Nov 03 00:43:33.146638 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:33.146428 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7961012-23c1-458a-83fe-0354645dc03a" containerName="revoke-keys" Nov 03 00:43:33.149463 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:33.149436 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7de5d31-zhthq" Nov 03 00:43:33.201719 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:33.201685 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:43:33.211547 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:33.211506 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd10eac-9837-4539-b484-f858468ef5cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7de5d31-zhthq\" (UID: \"0dd10eac-9837-4539-b484-f858468ef5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7de5d31-zhthq" Nov 03 00:43:33.211750 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:33.211605 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd10eac-9837-4539-b484-f858468ef5cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7de5d31-zhthq\" (UID: \"0dd10eac-9837-4539-b484-f858468ef5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7de5d31-zhthq" Nov 03 00:43:33.211750 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:33.211704 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nghd\" (UniqueName: \"kubernetes.io/projected/0dd10eac-9837-4539-b484-f858468ef5cf-kube-api-access-6nghd\") pod \"maas-api-revoke-sub-c7de5d31-zhthq\" (UID: \"0dd10eac-9837-4539-b484-f858468ef5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7de5d31-zhthq" Nov 03 00:43:33.251313 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:33.251276 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:43:33.251313 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:33.251276 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:43:33.251592 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:33.251334 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:43:33.252709 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:33.252690 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7de5d31-zhthq"] Nov 03 00:43:33.312985 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:33.312943 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd10eac-9837-4539-b484-f858468ef5cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7de5d31-zhthq\" (UID: \"0dd10eac-9837-4539-b484-f858468ef5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7de5d31-zhthq" Nov 03 00:43:33.313209 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:33.312999 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd10eac-9837-4539-b484-f858468ef5cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7de5d31-zhthq\" (UID: \"0dd10eac-9837-4539-b484-f858468ef5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7de5d31-zhthq" Nov 03 00:43:33.313209 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:33.313088 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nghd\" (UniqueName: \"kubernetes.io/projected/0dd10eac-9837-4539-b484-f858468ef5cf-kube-api-access-6nghd\") pod \"maas-api-revoke-sub-c7de5d31-zhthq\" (UID: \"0dd10eac-9837-4539-b484-f858468ef5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7de5d31-zhthq" Nov 03 00:43:33.313811 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:33.313780 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd10eac-9837-4539-b484-f858468ef5cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7de5d31-zhthq\" (UID: \"0dd10eac-9837-4539-b484-f858468ef5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7de5d31-zhthq" Nov 03 00:43:33.315681 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:33.315658 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd10eac-9837-4539-b484-f858468ef5cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7de5d31-zhthq\" (UID: \"0dd10eac-9837-4539-b484-f858468ef5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7de5d31-zhthq" Nov 03 00:43:33.374369 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:33.374328 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nghd\" (UniqueName: \"kubernetes.io/projected/0dd10eac-9837-4539-b484-f858468ef5cf-kube-api-access-6nghd\") pod \"maas-api-revoke-sub-c7de5d31-zhthq\" (UID: \"0dd10eac-9837-4539-b484-f858468ef5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7de5d31-zhthq" Nov 03 00:43:33.463522 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:33.463408 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7de5d31-zhthq" Nov 03 00:43:33.836209 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:33.836170 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7de5d31-zhthq"] Nov 03 00:43:34.314808 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:34.314763 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7de5d31-zhthq" event={"ID":"0dd10eac-9837-4539-b484-f858468ef5cf","Type":"ContainerStarted","Data":"17fe7023bac905e42d2cb1b8d88c0084416b60f1f19e215e0980bc22722298c9"} Nov 03 00:43:34.315195 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:34.314838 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7de5d31-zhthq" event={"ID":"0dd10eac-9837-4539-b484-f858468ef5cf","Type":"ContainerStarted","Data":"a0338689c16d4aaa9d97fc3c6659b973ab7a3c73a9cdb3dff17f20262d1c4892"} Nov 03 00:43:34.429892 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:34.429768 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7de5d31-zhthq" podStartSLOduration=1.42974871 podStartE2EDuration="1.42974871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:43:34.428887342 +0000 UTC m=+6889.781920732" watchObservedRunningTime="2026-11-03 00:43:34.42974871 +0000 UTC m=+6889.782782087" Nov 03 00:43:35.321063 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:35.321024 5114 generic.go:356] "Generic (PLEG): container finished" podID="0dd10eac-9837-4539-b484-f858468ef5cf" containerID="17fe7023bac905e42d2cb1b8d88c0084416b60f1f19e215e0980bc22722298c9" exitCode=0 Nov 03 00:43:35.321461 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:35.321108 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7de5d31-zhthq" event={"ID":"0dd10eac-9837-4539-b484-f858468ef5cf","Type":"ContainerDied","Data":"17fe7023bac905e42d2cb1b8d88c0084416b60f1f19e215e0980bc22722298c9"} Nov 03 00:43:36.463222 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:36.463190 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7de5d31-zhthq" Nov 03 00:43:36.544993 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:36.544953 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dd10eac-9837-4539-b484-f858468ef5cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd10eac-9837-4539-b484-f858468ef5cf-maas-api-xks-ca\") pod \"0dd10eac-9837-4539-b484-f858468ef5cf\" (UID: \"0dd10eac-9837-4539-b484-f858468ef5cf\") " Nov 03 00:43:36.544993 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:36.544996 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dd10eac-9837-4539-b484-f858468ef5cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd10eac-9837-4539-b484-f858468ef5cf-maas-api-openshift-ca\") pod \"0dd10eac-9837-4539-b484-f858468ef5cf\" (UID: \"0dd10eac-9837-4539-b484-f858468ef5cf\") " Nov 03 00:43:36.545248 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:36.545052 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd10eac-9837-4539-b484-f858468ef5cf-kube-api-access-6nghd\" (UniqueName: \"kubernetes.io/projected/0dd10eac-9837-4539-b484-f858468ef5cf-kube-api-access-6nghd\") pod \"0dd10eac-9837-4539-b484-f858468ef5cf\" (UID: \"0dd10eac-9837-4539-b484-f858468ef5cf\") " Nov 03 00:43:36.545455 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:36.545431 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd10eac-9837-4539-b484-f858468ef5cf-maas-api-openshift-ca" pod "0dd10eac-9837-4539-b484-f858468ef5cf" (UID: "0dd10eac-9837-4539-b484-f858468ef5cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:43:36.547390 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:36.547363 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dd10eac-9837-4539-b484-f858468ef5cf-maas-api-xks-ca" pod "0dd10eac-9837-4539-b484-f858468ef5cf" (UID: "0dd10eac-9837-4539-b484-f858468ef5cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:43:36.547390 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:36.547380 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd10eac-9837-4539-b484-f858468ef5cf-kube-api-access-6nghd" pod "0dd10eac-9837-4539-b484-f858468ef5cf" (UID: "0dd10eac-9837-4539-b484-f858468ef5cf"). InnerVolumeSpecName "kube-api-access-6nghd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:43:36.646355 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:36.646262 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nghd\" (UniqueName: \"kubernetes.io/projected/0dd10eac-9837-4539-b484-f858468ef5cf-kube-api-access-6nghd\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:36.646355 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:36.646300 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd10eac-9837-4539-b484-f858468ef5cf-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:36.646355 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:36.646310 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd10eac-9837-4539-b484-f858468ef5cf-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:37.331872 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:37.331839 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7de5d31-zhthq" Nov 03 00:43:37.331872 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:37.331845 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7de5d31-zhthq" event={"ID":"0dd10eac-9837-4539-b484-f858468ef5cf","Type":"ContainerDied","Data":"a0338689c16d4aaa9d97fc3c6659b973ab7a3c73a9cdb3dff17f20262d1c4892"} Nov 03 00:43:37.332153 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:43:37.331890 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0338689c16d4aaa9d97fc3c6659b973ab7a3c73a9cdb3dff17f20262d1c4892" Nov 03 00:43:37.582060 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:43:37.581974 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:44:07.583218 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:44:07.583182 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:44:21.172624 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:21.172589 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b6f0b52-5vbg5"] Nov 03 00:44:21.173146 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:21.172928 5114 cpu_manager.go:414] "containerMap: removing container" podUID="0dd10eac-9837-4539-b484-f858468ef5cf" containerName="revoke-keys" Nov 03 00:44:21.173146 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:21.172938 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd10eac-9837-4539-b484-f858468ef5cf" containerName="revoke-keys" Nov 03 00:44:21.173146 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:21.172995 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd10eac-9837-4539-b484-f858468ef5cf" containerName="revoke-keys" Nov 03 00:44:21.176005 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:21.175985 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6f0b52-5vbg5" Nov 03 00:44:21.254701 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:21.254657 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfgsv\" (UniqueName: \"kubernetes.io/projected/d01e42a2-9e61-48b4-b304-030a63f67f0d-kube-api-access-nfgsv\") pod \"maas-api-revoke-sub-8b6f0b52-5vbg5\" (UID: \"d01e42a2-9e61-48b4-b304-030a63f67f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6f0b52-5vbg5" Nov 03 00:44:21.254902 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:21.254767 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d01e42a2-9e61-48b4-b304-030a63f67f0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b6f0b52-5vbg5\" (UID: \"d01e42a2-9e61-48b4-b304-030a63f67f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6f0b52-5vbg5" Nov 03 00:44:21.254902 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:21.254810 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d01e42a2-9e61-48b4-b304-030a63f67f0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b6f0b52-5vbg5\" (UID: \"d01e42a2-9e61-48b4-b304-030a63f67f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6f0b52-5vbg5" Nov 03 00:44:21.277562 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:21.277514 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:44:21.277562 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:21.277528 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:44:21.277562 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:21.277537 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:44:21.277562 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:21.277558 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:44:21.279116 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:21.279096 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b6f0b52-5vbg5"] Nov 03 00:44:21.356030 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:21.355999 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d01e42a2-9e61-48b4-b304-030a63f67f0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b6f0b52-5vbg5\" (UID: \"d01e42a2-9e61-48b4-b304-030a63f67f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6f0b52-5vbg5" Nov 03 00:44:21.356259 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:21.356045 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d01e42a2-9e61-48b4-b304-030a63f67f0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b6f0b52-5vbg5\" (UID: \"d01e42a2-9e61-48b4-b304-030a63f67f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6f0b52-5vbg5" Nov 03 00:44:21.356259 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:21.356095 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfgsv\" (UniqueName: \"kubernetes.io/projected/d01e42a2-9e61-48b4-b304-030a63f67f0d-kube-api-access-nfgsv\") pod \"maas-api-revoke-sub-8b6f0b52-5vbg5\" (UID: \"d01e42a2-9e61-48b4-b304-030a63f67f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6f0b52-5vbg5" Nov 03 00:44:21.356866 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:21.356839 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d01e42a2-9e61-48b4-b304-030a63f67f0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b6f0b52-5vbg5\" (UID: \"d01e42a2-9e61-48b4-b304-030a63f67f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6f0b52-5vbg5" Nov 03 00:44:21.358653 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:21.358629 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d01e42a2-9e61-48b4-b304-030a63f67f0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b6f0b52-5vbg5\" (UID: \"d01e42a2-9e61-48b4-b304-030a63f67f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6f0b52-5vbg5" Nov 03 00:44:21.416760 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:21.416723 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfgsv\" (UniqueName: \"kubernetes.io/projected/d01e42a2-9e61-48b4-b304-030a63f67f0d-kube-api-access-nfgsv\") pod \"maas-api-revoke-sub-8b6f0b52-5vbg5\" (UID: \"d01e42a2-9e61-48b4-b304-030a63f67f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6f0b52-5vbg5" Nov 03 00:44:21.490665 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:21.490576 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6f0b52-5vbg5" Nov 03 00:44:21.863110 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:21.863074 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b6f0b52-5vbg5"] Nov 03 00:44:21.866666 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:44:21.866638 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd01e42a2_9e61_48b4_b304_030a63f67f0d.slice/crio-a50af59f86271aadca1be74ec18138d557c577de1c3f435a42d884037a54ee0c WatchSource:0}: Error finding container a50af59f86271aadca1be74ec18138d557c577de1c3f435a42d884037a54ee0c: Status 404 returned error can't find the container with id a50af59f86271aadca1be74ec18138d557c577de1c3f435a42d884037a54ee0c Nov 03 00:44:22.615189 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:22.615145 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6f0b52-5vbg5" event={"ID":"d01e42a2-9e61-48b4-b304-030a63f67f0d","Type":"ContainerStarted","Data":"bcedf11361cfdd0fe7525d3cbf59d0e6fef1b0b7b1b3c94873418f11f01267fc"} Nov 03 00:44:22.615663 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:22.615216 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6f0b52-5vbg5" event={"ID":"d01e42a2-9e61-48b4-b304-030a63f67f0d","Type":"ContainerStarted","Data":"a50af59f86271aadca1be74ec18138d557c577de1c3f435a42d884037a54ee0c"} Nov 03 00:44:22.731288 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:22.731228 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6f0b52-5vbg5" podStartSLOduration=1.731214296 podStartE2EDuration="1.731214296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:22.729920147 +0000 UTC m=+6938.082953563" watchObservedRunningTime="2026-11-03 00:44:22.731214296 +0000 UTC m=+6938.084247671" Nov 03 00:44:23.621440 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:23.621402 5114 generic.go:356] "Generic (PLEG): container finished" podID="d01e42a2-9e61-48b4-b304-030a63f67f0d" containerID="bcedf11361cfdd0fe7525d3cbf59d0e6fef1b0b7b1b3c94873418f11f01267fc" exitCode=0 Nov 03 00:44:23.621941 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:23.621510 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6f0b52-5vbg5" event={"ID":"d01e42a2-9e61-48b4-b304-030a63f67f0d","Type":"ContainerDied","Data":"bcedf11361cfdd0fe7525d3cbf59d0e6fef1b0b7b1b3c94873418f11f01267fc"} Nov 03 00:44:24.757654 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:24.757619 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6f0b52-5vbg5" Nov 03 00:44:24.795958 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:24.795924 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d01e42a2-9e61-48b4-b304-030a63f67f0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d01e42a2-9e61-48b4-b304-030a63f67f0d-maas-api-openshift-ca\") pod \"d01e42a2-9e61-48b4-b304-030a63f67f0d\" (UID: \"d01e42a2-9e61-48b4-b304-030a63f67f0d\") " Nov 03 00:44:24.796158 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:24.795976 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d01e42a2-9e61-48b4-b304-030a63f67f0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d01e42a2-9e61-48b4-b304-030a63f67f0d-maas-api-xks-ca\") pod \"d01e42a2-9e61-48b4-b304-030a63f67f0d\" (UID: \"d01e42a2-9e61-48b4-b304-030a63f67f0d\") " Nov 03 00:44:24.796158 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:24.796064 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d01e42a2-9e61-48b4-b304-030a63f67f0d-kube-api-access-nfgsv\" (UniqueName: \"kubernetes.io/projected/d01e42a2-9e61-48b4-b304-030a63f67f0d-kube-api-access-nfgsv\") pod \"d01e42a2-9e61-48b4-b304-030a63f67f0d\" (UID: \"d01e42a2-9e61-48b4-b304-030a63f67f0d\") " Nov 03 00:44:24.796367 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:24.796336 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d01e42a2-9e61-48b4-b304-030a63f67f0d-maas-api-openshift-ca" pod "d01e42a2-9e61-48b4-b304-030a63f67f0d" (UID: "d01e42a2-9e61-48b4-b304-030a63f67f0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:44:24.798288 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:24.798260 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d01e42a2-9e61-48b4-b304-030a63f67f0d-maas-api-xks-ca" pod "d01e42a2-9e61-48b4-b304-030a63f67f0d" (UID: "d01e42a2-9e61-48b4-b304-030a63f67f0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:44:24.798417 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:24.798371 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d01e42a2-9e61-48b4-b304-030a63f67f0d-kube-api-access-nfgsv" pod "d01e42a2-9e61-48b4-b304-030a63f67f0d" (UID: "d01e42a2-9e61-48b4-b304-030a63f67f0d"). InnerVolumeSpecName "kube-api-access-nfgsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:44:24.897531 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:24.897422 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d01e42a2-9e61-48b4-b304-030a63f67f0d-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:24.897531 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:24.897463 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfgsv\" (UniqueName: \"kubernetes.io/projected/d01e42a2-9e61-48b4-b304-030a63f67f0d-kube-api-access-nfgsv\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:24.897531 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:24.897474 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d01e42a2-9e61-48b4-b304-030a63f67f0d-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:25.631648 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:25.631605 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6f0b52-5vbg5" event={"ID":"d01e42a2-9e61-48b4-b304-030a63f67f0d","Type":"ContainerDied","Data":"a50af59f86271aadca1be74ec18138d557c577de1c3f435a42d884037a54ee0c"} Nov 03 00:44:25.631648 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:25.631645 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6f0b52-5vbg5" Nov 03 00:44:25.631872 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:25.631656 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a50af59f86271aadca1be74ec18138d557c577de1c3f435a42d884037a54ee0c" Nov 03 00:44:31.932082 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:31.932047 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e265255-f5t8n"] Nov 03 00:44:31.932617 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:31.932531 5114 cpu_manager.go:414] "containerMap: removing container" podUID="d01e42a2-9e61-48b4-b304-030a63f67f0d" containerName="revoke-keys" Nov 03 00:44:31.932617 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:31.932549 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d01e42a2-9e61-48b4-b304-030a63f67f0d" containerName="revoke-keys" Nov 03 00:44:31.932751 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:31.932654 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="d01e42a2-9e61-48b4-b304-030a63f67f0d" containerName="revoke-keys" Nov 03 00:44:31.935839 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:31.935819 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e265255-f5t8n" Nov 03 00:44:32.037305 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:32.037268 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:44:32.037514 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:32.037305 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:44:32.037514 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:32.037305 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:44:32.037514 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:32.037318 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:44:32.038794 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:32.038775 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e265255-f5t8n"] Nov 03 00:44:32.059921 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:32.059875 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/743bbffc-bba5-4154-8f25-3aa2ce6784c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e265255-f5t8n\" (UID: \"743bbffc-bba5-4154-8f25-3aa2ce6784c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e265255-f5t8n" Nov 03 00:44:32.059921 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:32.059920 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88l8s\" (UniqueName: \"kubernetes.io/projected/743bbffc-bba5-4154-8f25-3aa2ce6784c3-kube-api-access-88l8s\") pod \"maas-api-revoke-sub-4e265255-f5t8n\" (UID: \"743bbffc-bba5-4154-8f25-3aa2ce6784c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e265255-f5t8n" Nov 03 00:44:32.060175 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:32.059941 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/743bbffc-bba5-4154-8f25-3aa2ce6784c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e265255-f5t8n\" (UID: \"743bbffc-bba5-4154-8f25-3aa2ce6784c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e265255-f5t8n" Nov 03 00:44:32.161475 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:32.161434 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/743bbffc-bba5-4154-8f25-3aa2ce6784c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e265255-f5t8n\" (UID: \"743bbffc-bba5-4154-8f25-3aa2ce6784c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e265255-f5t8n" Nov 03 00:44:32.161675 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:32.161518 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88l8s\" (UniqueName: \"kubernetes.io/projected/743bbffc-bba5-4154-8f25-3aa2ce6784c3-kube-api-access-88l8s\") pod \"maas-api-revoke-sub-4e265255-f5t8n\" (UID: \"743bbffc-bba5-4154-8f25-3aa2ce6784c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e265255-f5t8n" Nov 03 00:44:32.161675 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:32.161566 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/743bbffc-bba5-4154-8f25-3aa2ce6784c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e265255-f5t8n\" (UID: \"743bbffc-bba5-4154-8f25-3aa2ce6784c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e265255-f5t8n" Nov 03 00:44:32.162249 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:32.162228 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/743bbffc-bba5-4154-8f25-3aa2ce6784c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e265255-f5t8n\" (UID: \"743bbffc-bba5-4154-8f25-3aa2ce6784c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e265255-f5t8n" Nov 03 00:44:32.164058 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:32.164038 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/743bbffc-bba5-4154-8f25-3aa2ce6784c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e265255-f5t8n\" (UID: \"743bbffc-bba5-4154-8f25-3aa2ce6784c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e265255-f5t8n" Nov 03 00:44:32.222517 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:32.222405 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88l8s\" (UniqueName: \"kubernetes.io/projected/743bbffc-bba5-4154-8f25-3aa2ce6784c3-kube-api-access-88l8s\") pod \"maas-api-revoke-sub-4e265255-f5t8n\" (UID: \"743bbffc-bba5-4154-8f25-3aa2ce6784c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e265255-f5t8n" Nov 03 00:44:32.248452 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:32.248405 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e265255-f5t8n" Nov 03 00:44:32.619992 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:32.619959 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e265255-f5t8n"] Nov 03 00:44:32.663126 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:32.663090 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e265255-f5t8n" event={"ID":"743bbffc-bba5-4154-8f25-3aa2ce6784c3","Type":"ContainerStarted","Data":"865f425be9956295dbcb660900cf08c2fa7e8003ba354f15b14562acb675a182"} Nov 03 00:44:33.670152 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:33.670111 5114 generic.go:356] "Generic (PLEG): container finished" podID="743bbffc-bba5-4154-8f25-3aa2ce6784c3" containerID="b832382503572fd1b2dc401710f548d1c1f3a267c5a0234838ca6e7762738e57" exitCode=0 Nov 03 00:44:33.670611 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:33.670194 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e265255-f5t8n" event={"ID":"743bbffc-bba5-4154-8f25-3aa2ce6784c3","Type":"ContainerDied","Data":"b832382503572fd1b2dc401710f548d1c1f3a267c5a0234838ca6e7762738e57"} Nov 03 00:44:34.852634 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:34.852598 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e265255-f5t8n" Nov 03 00:44:34.885836 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:34.885802 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/743bbffc-bba5-4154-8f25-3aa2ce6784c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/743bbffc-bba5-4154-8f25-3aa2ce6784c3-maas-api-xks-ca\") pod \"743bbffc-bba5-4154-8f25-3aa2ce6784c3\" (UID: \"743bbffc-bba5-4154-8f25-3aa2ce6784c3\") " Nov 03 00:44:34.886052 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:34.885901 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/743bbffc-bba5-4154-8f25-3aa2ce6784c3-kube-api-access-88l8s\" (UniqueName: \"kubernetes.io/projected/743bbffc-bba5-4154-8f25-3aa2ce6784c3-kube-api-access-88l8s\") pod \"743bbffc-bba5-4154-8f25-3aa2ce6784c3\" (UID: \"743bbffc-bba5-4154-8f25-3aa2ce6784c3\") " Nov 03 00:44:34.886052 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:34.885943 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/743bbffc-bba5-4154-8f25-3aa2ce6784c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/743bbffc-bba5-4154-8f25-3aa2ce6784c3-maas-api-openshift-ca\") pod \"743bbffc-bba5-4154-8f25-3aa2ce6784c3\" (UID: \"743bbffc-bba5-4154-8f25-3aa2ce6784c3\") " Nov 03 00:44:34.886742 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:34.886419 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/743bbffc-bba5-4154-8f25-3aa2ce6784c3-maas-api-openshift-ca" pod "743bbffc-bba5-4154-8f25-3aa2ce6784c3" (UID: "743bbffc-bba5-4154-8f25-3aa2ce6784c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:44:34.886945 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:34.886906 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/743bbffc-bba5-4154-8f25-3aa2ce6784c3-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:34.889292 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:34.889265 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/743bbffc-bba5-4154-8f25-3aa2ce6784c3-kube-api-access-88l8s" pod "743bbffc-bba5-4154-8f25-3aa2ce6784c3" (UID: "743bbffc-bba5-4154-8f25-3aa2ce6784c3"). InnerVolumeSpecName "kube-api-access-88l8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:44:34.889522 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:34.889465 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/743bbffc-bba5-4154-8f25-3aa2ce6784c3-maas-api-xks-ca" pod "743bbffc-bba5-4154-8f25-3aa2ce6784c3" (UID: "743bbffc-bba5-4154-8f25-3aa2ce6784c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:44:34.987990 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:34.987890 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88l8s\" (UniqueName: \"kubernetes.io/projected/743bbffc-bba5-4154-8f25-3aa2ce6784c3-kube-api-access-88l8s\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:34.987990 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:34.987932 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/743bbffc-bba5-4154-8f25-3aa2ce6784c3-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:35.682993 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:35.682962 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e265255-f5t8n" event={"ID":"743bbffc-bba5-4154-8f25-3aa2ce6784c3","Type":"ContainerDied","Data":"865f425be9956295dbcb660900cf08c2fa7e8003ba354f15b14562acb675a182"} Nov 03 00:44:35.683230 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:35.683216 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="865f425be9956295dbcb660900cf08c2fa7e8003ba354f15b14562acb675a182" Nov 03 00:44:35.683345 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:44:35.683336 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e265255-f5t8n" Nov 03 00:44:37.582103 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:44:37.582061 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:45:00.168604 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:00.168561 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894445-h898t"] Nov 03 00:45:00.169139 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:00.168918 5114 cpu_manager.go:414] "containerMap: removing container" podUID="743bbffc-bba5-4154-8f25-3aa2ce6784c3" containerName="revoke-keys" Nov 03 00:45:00.169139 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:00.168929 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="743bbffc-bba5-4154-8f25-3aa2ce6784c3" containerName="revoke-keys" Nov 03 00:45:00.169139 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:00.168978 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="743bbffc-bba5-4154-8f25-3aa2ce6784c3" containerName="revoke-keys" Nov 03 00:45:00.172151 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:00.172124 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-h898t" Nov 03 00:45:00.227579 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:00.227546 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:45:00.227838 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:00.227549 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:45:00.256169 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:00.256129 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpmkp\" (UniqueName: \"kubernetes.io/projected/f4ab16f9-fd12-412d-948a-5373ade0d095-kube-api-access-dpmkp\") pod \"maas-api-key-cleanup-29894445-h898t\" (UID: \"f4ab16f9-fd12-412d-948a-5373ade0d095\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-h898t" Nov 03 00:45:00.256371 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:00.256187 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ab16f9-fd12-412d-948a-5373ade0d095-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894445-h898t\" (UID: \"f4ab16f9-fd12-412d-948a-5373ade0d095\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-h898t" Nov 03 00:45:00.276690 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:00.276663 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:45:00.278383 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:00.278354 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894445-h898t"] Nov 03 00:45:00.356890 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:00.356850 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpmkp\" (UniqueName: \"kubernetes.io/projected/f4ab16f9-fd12-412d-948a-5373ade0d095-kube-api-access-dpmkp\") pod \"maas-api-key-cleanup-29894445-h898t\" (UID: \"f4ab16f9-fd12-412d-948a-5373ade0d095\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-h898t" Nov 03 00:45:00.357101 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:00.356902 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ab16f9-fd12-412d-948a-5373ade0d095-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894445-h898t\" (UID: \"f4ab16f9-fd12-412d-948a-5373ade0d095\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-h898t" Nov 03 00:45:00.357541 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:00.357523 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ab16f9-fd12-412d-948a-5373ade0d095-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894445-h898t\" (UID: \"f4ab16f9-fd12-412d-948a-5373ade0d095\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-h898t" Nov 03 00:45:00.418048 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:00.418009 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpmkp\" (UniqueName: \"kubernetes.io/projected/f4ab16f9-fd12-412d-948a-5373ade0d095-kube-api-access-dpmkp\") pod \"maas-api-key-cleanup-29894445-h898t\" (UID: \"f4ab16f9-fd12-412d-948a-5373ade0d095\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-h898t" Nov 03 00:45:00.490430 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:00.490340 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-h898t" Nov 03 00:45:00.861137 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:00.861079 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894445-h898t"] Nov 03 00:45:00.863586 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:45:00.863557 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ab16f9_fd12_412d_948a_5373ade0d095.slice/crio-af24b188be1b02e1c2593cb93157408a7b8d1078eb7baca9d096696366b17d1c WatchSource:0}: Error finding container af24b188be1b02e1c2593cb93157408a7b8d1078eb7baca9d096696366b17d1c: Status 404 returned error can't find the container with id af24b188be1b02e1c2593cb93157408a7b8d1078eb7baca9d096696366b17d1c Nov 03 00:45:01.812382 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:01.810616 5114 generic.go:356] "Generic (PLEG): container finished" podID="f4ab16f9-fd12-412d-948a-5373ade0d095" containerID="531430ab90c08eb76d76471a42ed8349cfc25272d7165d4af8db4bbc9cf54df4" exitCode=60 Nov 03 00:45:01.812382 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:01.810694 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-h898t" event={"ID":"f4ab16f9-fd12-412d-948a-5373ade0d095","Type":"ContainerDied","Data":"531430ab90c08eb76d76471a42ed8349cfc25272d7165d4af8db4bbc9cf54df4"} Nov 03 00:45:01.812382 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:01.810763 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-h898t" event={"ID":"f4ab16f9-fd12-412d-948a-5373ade0d095","Type":"ContainerStarted","Data":"af24b188be1b02e1c2593cb93157408a7b8d1078eb7baca9d096696366b17d1c"} Nov 03 00:45:01.812382 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:01.811243 5114 scope.go:122] "RemoveContainer" containerID="531430ab90c08eb76d76471a42ed8349cfc25272d7165d4af8db4bbc9cf54df4" Nov 03 00:45:02.817475 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:02.817443 5114 generic.go:356] "Generic (PLEG): container finished" podID="f4ab16f9-fd12-412d-948a-5373ade0d095" containerID="72c40712da005de8d765ca1a20d6e27144b98af329cb8d890eb70d0b571af797" exitCode=60 Nov 03 00:45:02.817950 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:02.817511 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-h898t" event={"ID":"f4ab16f9-fd12-412d-948a-5373ade0d095","Type":"ContainerDied","Data":"72c40712da005de8d765ca1a20d6e27144b98af329cb8d890eb70d0b571af797"} Nov 03 00:45:02.817950 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:02.817588 5114 scope.go:122] "RemoveContainer" containerID="531430ab90c08eb76d76471a42ed8349cfc25272d7165d4af8db4bbc9cf54df4" Nov 03 00:45:02.817950 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:02.817834 5114 scope.go:122] "RemoveContainer" containerID="72c40712da005de8d765ca1a20d6e27144b98af329cb8d890eb70d0b571af797" Nov 03 00:45:02.818079 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:45:02.818060 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894445-h898t_odh-ai-gateway-infra(f4ab16f9-fd12-412d-948a-5373ade0d095)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-h898t" podUID="f4ab16f9-fd12-412d-948a-5373ade0d095" Nov 03 00:45:03.824410 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:03.824374 5114 scope.go:122] "RemoveContainer" containerID="72c40712da005de8d765ca1a20d6e27144b98af329cb8d890eb70d0b571af797" Nov 03 00:45:03.824929 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:45:03.824714 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894445-h898t_odh-ai-gateway-infra(f4ab16f9-fd12-412d-948a-5373ade0d095)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-h898t" podUID="f4ab16f9-fd12-412d-948a-5373ade0d095" Nov 03 00:45:07.583309 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:45:07.583268 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:45:16.787837 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:16.787801 5114 scope.go:122] "RemoveContainer" containerID="72c40712da005de8d765ca1a20d6e27144b98af329cb8d890eb70d0b571af797" Nov 03 00:45:17.919665 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:17.919617 5114 generic.go:356] "Generic (PLEG): container finished" podID="f4ab16f9-fd12-412d-948a-5373ade0d095" containerID="fe1393c7c55c82dc33ceac922b52b40cb3866bf07733f458c6c4b030eaf5ab3e" exitCode=60 Nov 03 00:45:17.920383 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:17.919766 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-h898t" event={"ID":"f4ab16f9-fd12-412d-948a-5373ade0d095","Type":"ContainerDied","Data":"fe1393c7c55c82dc33ceac922b52b40cb3866bf07733f458c6c4b030eaf5ab3e"} Nov 03 00:45:17.921190 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:17.921171 5114 scope.go:122] "RemoveContainer" containerID="72c40712da005de8d765ca1a20d6e27144b98af329cb8d890eb70d0b571af797" Nov 03 00:45:17.923924 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:17.923749 5114 scope.go:122] "RemoveContainer" containerID="fe1393c7c55c82dc33ceac922b52b40cb3866bf07733f458c6c4b030eaf5ab3e" Nov 03 00:45:17.924156 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:45:17.924044 5114 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894445-h898t_odh-ai-gateway-infra(f4ab16f9-fd12-412d-948a-5373ade0d095)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-h898t" podUID="f4ab16f9-fd12-412d-948a-5373ade0d095" Nov 03 00:45:19.047585 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:19.047546 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894445-h898t"] Nov 03 00:45:19.177655 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:19.177624 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-h898t" Nov 03 00:45:19.188785 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:19.188745 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4ab16f9-fd12-412d-948a-5373ade0d095-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ab16f9-fd12-412d-948a-5373ade0d095-maas-api-service-ca\") pod \"f4ab16f9-fd12-412d-948a-5373ade0d095\" (UID: \"f4ab16f9-fd12-412d-948a-5373ade0d095\") " Nov 03 00:45:19.189021 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:19.188844 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4ab16f9-fd12-412d-948a-5373ade0d095-kube-api-access-dpmkp\" (UniqueName: \"kubernetes.io/projected/f4ab16f9-fd12-412d-948a-5373ade0d095-kube-api-access-dpmkp\") pod \"f4ab16f9-fd12-412d-948a-5373ade0d095\" (UID: \"f4ab16f9-fd12-412d-948a-5373ade0d095\") " Nov 03 00:45:19.189155 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:19.189132 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ab16f9-fd12-412d-948a-5373ade0d095-maas-api-service-ca" pod "f4ab16f9-fd12-412d-948a-5373ade0d095" (UID: "f4ab16f9-fd12-412d-948a-5373ade0d095"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:45:19.191260 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:19.191234 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ab16f9-fd12-412d-948a-5373ade0d095-kube-api-access-dpmkp" pod "f4ab16f9-fd12-412d-948a-5373ade0d095" (UID: "f4ab16f9-fd12-412d-948a-5373ade0d095"). InnerVolumeSpecName "kube-api-access-dpmkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:45:19.289572 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:19.289533 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ab16f9-fd12-412d-948a-5373ade0d095-maas-api-service-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:19.289572 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:19.289565 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpmkp\" (UniqueName: \"kubernetes.io/projected/f4ab16f9-fd12-412d-948a-5373ade0d095-kube-api-access-dpmkp\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:19.935571 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:19.935541 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-h898t" Nov 03 00:45:19.935761 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:19.935567 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-h898t" event={"ID":"f4ab16f9-fd12-412d-948a-5373ade0d095","Type":"ContainerDied","Data":"af24b188be1b02e1c2593cb93157408a7b8d1078eb7baca9d096696366b17d1c"} Nov 03 00:45:19.935761 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:19.935641 5114 scope.go:122] "RemoveContainer" containerID="fe1393c7c55c82dc33ceac922b52b40cb3866bf07733f458c6c4b030eaf5ab3e" Nov 03 00:45:20.097787 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:20.097745 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894445-h898t"] Nov 03 00:45:20.102679 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:20.102645 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894445-h898t"] Nov 03 00:45:20.792047 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:20.792013 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4ab16f9-fd12-412d-948a-5373ade0d095" path="/var/lib/kubelet/pods/f4ab16f9-fd12-412d-948a-5373ade0d095/volumes" Nov 03 00:45:20.963219 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:20.963179 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ce57900-d2x7m"] Nov 03 00:45:20.963549 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:20.963533 5114 cpu_manager.go:414] "containerMap: removing container" podUID="f4ab16f9-fd12-412d-948a-5373ade0d095" containerName="cleanup" Nov 03 00:45:20.963611 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:20.963552 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ab16f9-fd12-412d-948a-5373ade0d095" containerName="cleanup" Nov 03 00:45:20.963611 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:20.963580 5114 cpu_manager.go:414] "containerMap: removing container" podUID="f4ab16f9-fd12-412d-948a-5373ade0d095" containerName="cleanup" Nov 03 00:45:20.963611 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:20.963586 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ab16f9-fd12-412d-948a-5373ade0d095" containerName="cleanup" Nov 03 00:45:20.963714 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:20.963680 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4ab16f9-fd12-412d-948a-5373ade0d095" containerName="cleanup" Nov 03 00:45:20.963714 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:20.963693 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4ab16f9-fd12-412d-948a-5373ade0d095" containerName="cleanup" Nov 03 00:45:20.968100 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:20.968082 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce57900-d2x7m" Nov 03 00:45:21.004045 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:21.004001 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7445cbe5-60fd-4c6e-a2a7-19161eac085a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ce57900-d2x7m\" (UID: \"7445cbe5-60fd-4c6e-a2a7-19161eac085a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce57900-d2x7m" Nov 03 00:45:21.004045 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:21.004042 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wppz2\" (UniqueName: \"kubernetes.io/projected/7445cbe5-60fd-4c6e-a2a7-19161eac085a-kube-api-access-wppz2\") pod \"maas-api-revoke-sub-2ce57900-d2x7m\" (UID: \"7445cbe5-60fd-4c6e-a2a7-19161eac085a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce57900-d2x7m" Nov 03 00:45:21.004272 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:21.004066 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7445cbe5-60fd-4c6e-a2a7-19161eac085a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ce57900-d2x7m\" (UID: \"7445cbe5-60fd-4c6e-a2a7-19161eac085a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce57900-d2x7m" Nov 03 00:45:21.022799 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:21.022765 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:45:21.024545 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:21.024511 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ce57900-d2x7m"] Nov 03 00:45:21.071814 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:21.071711 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:45:21.072020 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:21.071813 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:45:21.072099 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:21.072020 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:45:21.105604 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:21.105567 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7445cbe5-60fd-4c6e-a2a7-19161eac085a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ce57900-d2x7m\" (UID: \"7445cbe5-60fd-4c6e-a2a7-19161eac085a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce57900-d2x7m" Nov 03 00:45:21.105604 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:21.105605 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wppz2\" (UniqueName: \"kubernetes.io/projected/7445cbe5-60fd-4c6e-a2a7-19161eac085a-kube-api-access-wppz2\") pod \"maas-api-revoke-sub-2ce57900-d2x7m\" (UID: \"7445cbe5-60fd-4c6e-a2a7-19161eac085a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce57900-d2x7m" Nov 03 00:45:21.106145 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:21.105644 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7445cbe5-60fd-4c6e-a2a7-19161eac085a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ce57900-d2x7m\" (UID: \"7445cbe5-60fd-4c6e-a2a7-19161eac085a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce57900-d2x7m" Nov 03 00:45:21.106396 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:21.106378 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7445cbe5-60fd-4c6e-a2a7-19161eac085a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ce57900-d2x7m\" (UID: \"7445cbe5-60fd-4c6e-a2a7-19161eac085a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce57900-d2x7m" Nov 03 00:45:21.108260 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:21.108238 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7445cbe5-60fd-4c6e-a2a7-19161eac085a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ce57900-d2x7m\" (UID: \"7445cbe5-60fd-4c6e-a2a7-19161eac085a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce57900-d2x7m" Nov 03 00:45:21.168130 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:21.168097 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wppz2\" (UniqueName: \"kubernetes.io/projected/7445cbe5-60fd-4c6e-a2a7-19161eac085a-kube-api-access-wppz2\") pod \"maas-api-revoke-sub-2ce57900-d2x7m\" (UID: \"7445cbe5-60fd-4c6e-a2a7-19161eac085a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce57900-d2x7m" Nov 03 00:45:21.280617 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:21.280578 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce57900-d2x7m" Nov 03 00:45:21.652011 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:21.651824 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ce57900-d2x7m"] Nov 03 00:45:21.654564 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:45:21.654528 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7445cbe5_60fd_4c6e_a2a7_19161eac085a.slice/crio-4761855e5b8abe33923259537a6b58b18f5b143bc9edfa832d8bf92bd2d5af2a WatchSource:0}: Error finding container 4761855e5b8abe33923259537a6b58b18f5b143bc9edfa832d8bf92bd2d5af2a: Status 404 returned error can't find the container with id 4761855e5b8abe33923259537a6b58b18f5b143bc9edfa832d8bf92bd2d5af2a Nov 03 00:45:21.946766 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:21.946726 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce57900-d2x7m" event={"ID":"7445cbe5-60fd-4c6e-a2a7-19161eac085a","Type":"ContainerStarted","Data":"105baedda4b0d7243caaa917615830c9aa8ce158ecdac560d6f8390aa32618dd"} Nov 03 00:45:21.946992 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:21.946788 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce57900-d2x7m" event={"ID":"7445cbe5-60fd-4c6e-a2a7-19161eac085a","Type":"ContainerStarted","Data":"4761855e5b8abe33923259537a6b58b18f5b143bc9edfa832d8bf92bd2d5af2a"} Nov 03 00:45:22.062474 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:22.062420 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce57900-d2x7m" podStartSLOduration=2.062403486 podStartE2EDuration="2.062403486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:22.061205661 +0000 UTC m=+6997.414239038" watchObservedRunningTime="2026-11-03 00:45:22.062403486 +0000 UTC m=+6997.415436899" Nov 03 00:45:22.953172 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:22.953134 5114 generic.go:356] "Generic (PLEG): container finished" podID="7445cbe5-60fd-4c6e-a2a7-19161eac085a" containerID="105baedda4b0d7243caaa917615830c9aa8ce158ecdac560d6f8390aa32618dd" exitCode=0 Nov 03 00:45:22.953609 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:22.953200 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce57900-d2x7m" event={"ID":"7445cbe5-60fd-4c6e-a2a7-19161eac085a","Type":"ContainerDied","Data":"105baedda4b0d7243caaa917615830c9aa8ce158ecdac560d6f8390aa32618dd"} Nov 03 00:45:24.091055 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:24.091022 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce57900-d2x7m" Nov 03 00:45:24.134210 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:24.134172 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7445cbe5-60fd-4c6e-a2a7-19161eac085a-kube-api-access-wppz2\" (UniqueName: \"kubernetes.io/projected/7445cbe5-60fd-4c6e-a2a7-19161eac085a-kube-api-access-wppz2\") pod \"7445cbe5-60fd-4c6e-a2a7-19161eac085a\" (UID: \"7445cbe5-60fd-4c6e-a2a7-19161eac085a\") " Nov 03 00:45:24.134377 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:24.134234 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7445cbe5-60fd-4c6e-a2a7-19161eac085a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7445cbe5-60fd-4c6e-a2a7-19161eac085a-maas-api-openshift-ca\") pod \"7445cbe5-60fd-4c6e-a2a7-19161eac085a\" (UID: \"7445cbe5-60fd-4c6e-a2a7-19161eac085a\") " Nov 03 00:45:24.134377 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:24.134323 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7445cbe5-60fd-4c6e-a2a7-19161eac085a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7445cbe5-60fd-4c6e-a2a7-19161eac085a-maas-api-xks-ca\") pod \"7445cbe5-60fd-4c6e-a2a7-19161eac085a\" (UID: \"7445cbe5-60fd-4c6e-a2a7-19161eac085a\") " Nov 03 00:45:24.134665 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:24.134634 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7445cbe5-60fd-4c6e-a2a7-19161eac085a-maas-api-openshift-ca" pod "7445cbe5-60fd-4c6e-a2a7-19161eac085a" (UID: "7445cbe5-60fd-4c6e-a2a7-19161eac085a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:45:24.136648 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:24.136624 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7445cbe5-60fd-4c6e-a2a7-19161eac085a-kube-api-access-wppz2" pod "7445cbe5-60fd-4c6e-a2a7-19161eac085a" (UID: "7445cbe5-60fd-4c6e-a2a7-19161eac085a"). InnerVolumeSpecName "kube-api-access-wppz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:45:24.136791 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:24.136707 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7445cbe5-60fd-4c6e-a2a7-19161eac085a-maas-api-xks-ca" pod "7445cbe5-60fd-4c6e-a2a7-19161eac085a" (UID: "7445cbe5-60fd-4c6e-a2a7-19161eac085a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:45:24.235746 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:24.235663 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wppz2\" (UniqueName: \"kubernetes.io/projected/7445cbe5-60fd-4c6e-a2a7-19161eac085a-kube-api-access-wppz2\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:24.235746 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:24.235695 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7445cbe5-60fd-4c6e-a2a7-19161eac085a-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:24.235746 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:24.235705 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7445cbe5-60fd-4c6e-a2a7-19161eac085a-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:24.963618 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:24.963585 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce57900-d2x7m" Nov 03 00:45:24.963822 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:24.963620 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce57900-d2x7m" event={"ID":"7445cbe5-60fd-4c6e-a2a7-19161eac085a","Type":"ContainerDied","Data":"4761855e5b8abe33923259537a6b58b18f5b143bc9edfa832d8bf92bd2d5af2a"} Nov 03 00:45:24.963822 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:24.963681 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4761855e5b8abe33923259537a6b58b18f5b143bc9edfa832d8bf92bd2d5af2a" Nov 03 00:45:31.659677 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:31.659634 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee7cc6e5-2696t"] Nov 03 00:45:31.660135 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:31.659991 5114 cpu_manager.go:414] "containerMap: removing container" podUID="f4ab16f9-fd12-412d-948a-5373ade0d095" containerName="cleanup" Nov 03 00:45:31.660135 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:31.660005 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ab16f9-fd12-412d-948a-5373ade0d095" containerName="cleanup" Nov 03 00:45:31.660135 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:31.660036 5114 cpu_manager.go:414] "containerMap: removing container" podUID="7445cbe5-60fd-4c6e-a2a7-19161eac085a" containerName="revoke-keys" Nov 03 00:45:31.660135 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:31.660042 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7445cbe5-60fd-4c6e-a2a7-19161eac085a" containerName="revoke-keys" Nov 03 00:45:31.660135 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:31.660109 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="7445cbe5-60fd-4c6e-a2a7-19161eac085a" containerName="revoke-keys" Nov 03 00:45:31.660135 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:31.660123 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4ab16f9-fd12-412d-948a-5373ade0d095" containerName="cleanup" Nov 03 00:45:31.663514 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:31.663492 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7cc6e5-2696t" Nov 03 00:45:31.703115 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:31.703077 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba20182f-8631-473a-b67f-91067e66f621-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee7cc6e5-2696t\" (UID: \"ba20182f-8631-473a-b67f-91067e66f621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7cc6e5-2696t" Nov 03 00:45:31.703115 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:31.703126 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwbn6\" (UniqueName: \"kubernetes.io/projected/ba20182f-8631-473a-b67f-91067e66f621-kube-api-access-cwbn6\") pod \"maas-api-revoke-sub-ee7cc6e5-2696t\" (UID: \"ba20182f-8631-473a-b67f-91067e66f621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7cc6e5-2696t" Nov 03 00:45:31.703368 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:31.703153 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba20182f-8631-473a-b67f-91067e66f621-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee7cc6e5-2696t\" (UID: \"ba20182f-8631-473a-b67f-91067e66f621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7cc6e5-2696t" Nov 03 00:45:31.718754 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:31.718584 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:45:31.718754 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:31.718678 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:45:31.765237 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:31.765203 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:45:31.765585 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:31.765565 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:45:31.767195 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:31.767171 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee7cc6e5-2696t"] Nov 03 00:45:31.804520 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:31.804458 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba20182f-8631-473a-b67f-91067e66f621-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee7cc6e5-2696t\" (UID: \"ba20182f-8631-473a-b67f-91067e66f621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7cc6e5-2696t" Nov 03 00:45:31.804729 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:31.804533 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwbn6\" (UniqueName: \"kubernetes.io/projected/ba20182f-8631-473a-b67f-91067e66f621-kube-api-access-cwbn6\") pod \"maas-api-revoke-sub-ee7cc6e5-2696t\" (UID: \"ba20182f-8631-473a-b67f-91067e66f621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7cc6e5-2696t" Nov 03 00:45:31.804729 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:31.804559 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba20182f-8631-473a-b67f-91067e66f621-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee7cc6e5-2696t\" (UID: \"ba20182f-8631-473a-b67f-91067e66f621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7cc6e5-2696t" Nov 03 00:45:31.805271 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:31.805242 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba20182f-8631-473a-b67f-91067e66f621-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee7cc6e5-2696t\" (UID: \"ba20182f-8631-473a-b67f-91067e66f621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7cc6e5-2696t" Nov 03 00:45:31.807158 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:31.807129 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba20182f-8631-473a-b67f-91067e66f621-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee7cc6e5-2696t\" (UID: \"ba20182f-8631-473a-b67f-91067e66f621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7cc6e5-2696t" Nov 03 00:45:31.865496 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:31.865458 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwbn6\" (UniqueName: \"kubernetes.io/projected/ba20182f-8631-473a-b67f-91067e66f621-kube-api-access-cwbn6\") pod \"maas-api-revoke-sub-ee7cc6e5-2696t\" (UID: \"ba20182f-8631-473a-b67f-91067e66f621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7cc6e5-2696t" Nov 03 00:45:31.976680 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:31.976589 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7cc6e5-2696t" Nov 03 00:45:32.349782 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:32.349755 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee7cc6e5-2696t"] Nov 03 00:45:32.352435 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:45:32.352396 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba20182f_8631_473a_b67f_91067e66f621.slice/crio-bfe3f5d677b1dc6f2ec4de84ad64c671d420be1c5cc553d3fd374357af2600f5 WatchSource:0}: Error finding container bfe3f5d677b1dc6f2ec4de84ad64c671d420be1c5cc553d3fd374357af2600f5: Status 404 returned error can't find the container with id bfe3f5d677b1dc6f2ec4de84ad64c671d420be1c5cc553d3fd374357af2600f5 Nov 03 00:45:33.002762 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:33.002663 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7cc6e5-2696t" event={"ID":"ba20182f-8631-473a-b67f-91067e66f621","Type":"ContainerStarted","Data":"4305eea4f4cd10a6bf046dce01e3d712ad80905832bcbab6505b70ab3f34ff2b"} Nov 03 00:45:33.002762 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:33.002725 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7cc6e5-2696t" event={"ID":"ba20182f-8631-473a-b67f-91067e66f621","Type":"ContainerStarted","Data":"bfe3f5d677b1dc6f2ec4de84ad64c671d420be1c5cc553d3fd374357af2600f5"} Nov 03 00:45:33.119227 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:33.119175 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7cc6e5-2696t" podStartSLOduration=2.119159968 podStartE2EDuration="2.119159968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:33.118228575 +0000 UTC m=+7008.471261951" watchObservedRunningTime="2026-11-03 00:45:33.119159968 +0000 UTC m=+7008.472193342" Nov 03 00:45:34.008260 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:34.008220 5114 generic.go:356] "Generic (PLEG): container finished" podID="ba20182f-8631-473a-b67f-91067e66f621" containerID="4305eea4f4cd10a6bf046dce01e3d712ad80905832bcbab6505b70ab3f34ff2b" exitCode=0 Nov 03 00:45:34.008741 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:34.008304 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7cc6e5-2696t" event={"ID":"ba20182f-8631-473a-b67f-91067e66f621","Type":"ContainerDied","Data":"4305eea4f4cd10a6bf046dce01e3d712ad80905832bcbab6505b70ab3f34ff2b"} Nov 03 00:45:35.146292 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:35.146258 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7cc6e5-2696t" Nov 03 00:45:35.234801 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:35.234766 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba20182f-8631-473a-b67f-91067e66f621-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba20182f-8631-473a-b67f-91067e66f621-maas-api-xks-ca\") pod \"ba20182f-8631-473a-b67f-91067e66f621\" (UID: \"ba20182f-8631-473a-b67f-91067e66f621\") " Nov 03 00:45:35.234801 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:35.234805 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba20182f-8631-473a-b67f-91067e66f621-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba20182f-8631-473a-b67f-91067e66f621-maas-api-openshift-ca\") pod \"ba20182f-8631-473a-b67f-91067e66f621\" (UID: \"ba20182f-8631-473a-b67f-91067e66f621\") " Nov 03 00:45:35.235045 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:35.234855 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba20182f-8631-473a-b67f-91067e66f621-kube-api-access-cwbn6\" (UniqueName: \"kubernetes.io/projected/ba20182f-8631-473a-b67f-91067e66f621-kube-api-access-cwbn6\") pod \"ba20182f-8631-473a-b67f-91067e66f621\" (UID: \"ba20182f-8631-473a-b67f-91067e66f621\") " Nov 03 00:45:35.235307 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:35.235264 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba20182f-8631-473a-b67f-91067e66f621-maas-api-openshift-ca" pod "ba20182f-8631-473a-b67f-91067e66f621" (UID: "ba20182f-8631-473a-b67f-91067e66f621"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:45:35.237149 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:35.237120 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba20182f-8631-473a-b67f-91067e66f621-maas-api-xks-ca" pod "ba20182f-8631-473a-b67f-91067e66f621" (UID: "ba20182f-8631-473a-b67f-91067e66f621"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:45:35.237227 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:35.237174 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba20182f-8631-473a-b67f-91067e66f621-kube-api-access-cwbn6" pod "ba20182f-8631-473a-b67f-91067e66f621" (UID: "ba20182f-8631-473a-b67f-91067e66f621"). InnerVolumeSpecName "kube-api-access-cwbn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:45:35.336426 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:35.336385 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba20182f-8631-473a-b67f-91067e66f621-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:35.336426 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:35.336421 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba20182f-8631-473a-b67f-91067e66f621-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:35.336426 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:35.336431 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwbn6\" (UniqueName: \"kubernetes.io/projected/ba20182f-8631-473a-b67f-91067e66f621-kube-api-access-cwbn6\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:36.019973 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:36.019934 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7cc6e5-2696t" event={"ID":"ba20182f-8631-473a-b67f-91067e66f621","Type":"ContainerDied","Data":"bfe3f5d677b1dc6f2ec4de84ad64c671d420be1c5cc553d3fd374357af2600f5"} Nov 03 00:45:36.020160 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:36.019985 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfe3f5d677b1dc6f2ec4de84ad64c671d420be1c5cc553d3fd374357af2600f5" Nov 03 00:45:36.020160 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:36.020006 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7cc6e5-2696t" Nov 03 00:45:37.584866 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:45:37.584748 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:45:42.563026 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:42.562993 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7a31d8e-8dgpv"] Nov 03 00:45:42.563544 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:42.563331 5114 cpu_manager.go:414] "containerMap: removing container" podUID="ba20182f-8631-473a-b67f-91067e66f621" containerName="revoke-keys" Nov 03 00:45:42.563544 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:42.563342 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba20182f-8631-473a-b67f-91067e66f621" containerName="revoke-keys" Nov 03 00:45:42.563544 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:42.563396 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba20182f-8631-473a-b67f-91067e66f621" containerName="revoke-keys" Nov 03 00:45:42.566299 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:42.566271 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a31d8e-8dgpv" Nov 03 00:45:42.618991 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:42.618957 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:45:42.668307 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:42.668263 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:45:42.668307 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:42.668300 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:45:42.668574 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:42.668299 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:45:42.670148 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:42.670120 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7a31d8e-8dgpv"] Nov 03 00:45:42.702075 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:42.702034 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzw5g\" (UniqueName: \"kubernetes.io/projected/bb802929-b2d0-4191-9f35-4cfee6005793-kube-api-access-rzw5g\") pod \"maas-api-revoke-sub-e7a31d8e-8dgpv\" (UID: \"bb802929-b2d0-4191-9f35-4cfee6005793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a31d8e-8dgpv" Nov 03 00:45:42.702075 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:42.702081 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb802929-b2d0-4191-9f35-4cfee6005793-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7a31d8e-8dgpv\" (UID: \"bb802929-b2d0-4191-9f35-4cfee6005793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a31d8e-8dgpv" Nov 03 00:45:42.702368 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:42.702215 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb802929-b2d0-4191-9f35-4cfee6005793-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7a31d8e-8dgpv\" (UID: \"bb802929-b2d0-4191-9f35-4cfee6005793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a31d8e-8dgpv" Nov 03 00:45:42.803789 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:42.803740 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzw5g\" (UniqueName: \"kubernetes.io/projected/bb802929-b2d0-4191-9f35-4cfee6005793-kube-api-access-rzw5g\") pod \"maas-api-revoke-sub-e7a31d8e-8dgpv\" (UID: \"bb802929-b2d0-4191-9f35-4cfee6005793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a31d8e-8dgpv" Nov 03 00:45:42.803789 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:42.803792 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb802929-b2d0-4191-9f35-4cfee6005793-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7a31d8e-8dgpv\" (UID: \"bb802929-b2d0-4191-9f35-4cfee6005793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a31d8e-8dgpv" Nov 03 00:45:42.804074 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:42.803845 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb802929-b2d0-4191-9f35-4cfee6005793-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7a31d8e-8dgpv\" (UID: \"bb802929-b2d0-4191-9f35-4cfee6005793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a31d8e-8dgpv" Nov 03 00:45:42.804609 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:42.804583 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb802929-b2d0-4191-9f35-4cfee6005793-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7a31d8e-8dgpv\" (UID: \"bb802929-b2d0-4191-9f35-4cfee6005793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a31d8e-8dgpv" Nov 03 00:45:42.806427 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:42.806408 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb802929-b2d0-4191-9f35-4cfee6005793-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7a31d8e-8dgpv\" (UID: \"bb802929-b2d0-4191-9f35-4cfee6005793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a31d8e-8dgpv" Nov 03 00:45:42.864538 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:42.864446 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzw5g\" (UniqueName: \"kubernetes.io/projected/bb802929-b2d0-4191-9f35-4cfee6005793-kube-api-access-rzw5g\") pod \"maas-api-revoke-sub-e7a31d8e-8dgpv\" (UID: \"bb802929-b2d0-4191-9f35-4cfee6005793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a31d8e-8dgpv" Nov 03 00:45:42.879337 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:42.879295 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a31d8e-8dgpv" Nov 03 00:45:43.254472 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:43.254439 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7a31d8e-8dgpv"] Nov 03 00:45:43.257005 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:45:43.256963 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb802929_b2d0_4191_9f35_4cfee6005793.slice/crio-c677b91acc9337b23516ef8080181509878a5abea8ca4c9b8e755dcf26913beb WatchSource:0}: Error finding container c677b91acc9337b23516ef8080181509878a5abea8ca4c9b8e755dcf26913beb: Status 404 returned error can't find the container with id c677b91acc9337b23516ef8080181509878a5abea8ca4c9b8e755dcf26913beb Nov 03 00:45:44.056910 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:44.056875 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a31d8e-8dgpv" event={"ID":"bb802929-b2d0-4191-9f35-4cfee6005793","Type":"ContainerStarted","Data":"bd8e9435d513d319963a4fcaf0ec9e2de1974293c1f5325b8a8258f0db1c6e81"} Nov 03 00:45:44.057358 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:44.056929 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a31d8e-8dgpv" event={"ID":"bb802929-b2d0-4191-9f35-4cfee6005793","Type":"ContainerStarted","Data":"c677b91acc9337b23516ef8080181509878a5abea8ca4c9b8e755dcf26913beb"} Nov 03 00:45:44.173505 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:44.173428 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a31d8e-8dgpv" podStartSLOduration=2.17341209 podStartE2EDuration="2.17341209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:44.172321945 +0000 UTC m=+7019.525355320" watchObservedRunningTime="2026-11-03 00:45:44.17341209 +0000 UTC m=+7019.526445466" Nov 03 00:45:45.065512 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:45.065451 5114 generic.go:356] "Generic (PLEG): container finished" podID="bb802929-b2d0-4191-9f35-4cfee6005793" containerID="bd8e9435d513d319963a4fcaf0ec9e2de1974293c1f5325b8a8258f0db1c6e81" exitCode=0 Nov 03 00:45:45.066007 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:45.065537 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a31d8e-8dgpv" event={"ID":"bb802929-b2d0-4191-9f35-4cfee6005793","Type":"ContainerDied","Data":"bd8e9435d513d319963a4fcaf0ec9e2de1974293c1f5325b8a8258f0db1c6e81"} Nov 03 00:45:46.404508 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:46.404463 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a31d8e-8dgpv" Nov 03 00:45:46.562057 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:46.561994 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb802929-b2d0-4191-9f35-4cfee6005793-kube-api-access-rzw5g\" (UniqueName: \"kubernetes.io/projected/bb802929-b2d0-4191-9f35-4cfee6005793-kube-api-access-rzw5g\") pod \"bb802929-b2d0-4191-9f35-4cfee6005793\" (UID: \"bb802929-b2d0-4191-9f35-4cfee6005793\") " Nov 03 00:45:46.562408 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:46.562366 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb802929-b2d0-4191-9f35-4cfee6005793-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb802929-b2d0-4191-9f35-4cfee6005793-maas-api-openshift-ca\") pod \"bb802929-b2d0-4191-9f35-4cfee6005793\" (UID: \"bb802929-b2d0-4191-9f35-4cfee6005793\") " Nov 03 00:45:46.562706 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:46.562685 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb802929-b2d0-4191-9f35-4cfee6005793-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb802929-b2d0-4191-9f35-4cfee6005793-maas-api-xks-ca\") pod \"bb802929-b2d0-4191-9f35-4cfee6005793\" (UID: \"bb802929-b2d0-4191-9f35-4cfee6005793\") " Nov 03 00:45:46.569520 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:46.567855 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb802929-b2d0-4191-9f35-4cfee6005793-maas-api-openshift-ca" pod "bb802929-b2d0-4191-9f35-4cfee6005793" (UID: "bb802929-b2d0-4191-9f35-4cfee6005793"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:45:46.589633 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:46.589561 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb802929-b2d0-4191-9f35-4cfee6005793-maas-api-xks-ca" pod "bb802929-b2d0-4191-9f35-4cfee6005793" (UID: "bb802929-b2d0-4191-9f35-4cfee6005793"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:45:46.590024 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:46.589974 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb802929-b2d0-4191-9f35-4cfee6005793-kube-api-access-rzw5g" pod "bb802929-b2d0-4191-9f35-4cfee6005793" (UID: "bb802929-b2d0-4191-9f35-4cfee6005793"). InnerVolumeSpecName "kube-api-access-rzw5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:45:46.668162 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:46.667997 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb802929-b2d0-4191-9f35-4cfee6005793-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:46.668162 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:46.668051 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzw5g\" (UniqueName: \"kubernetes.io/projected/bb802929-b2d0-4191-9f35-4cfee6005793-kube-api-access-rzw5g\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:46.668162 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:46.668069 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb802929-b2d0-4191-9f35-4cfee6005793-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:47.081899 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:47.081855 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a31d8e-8dgpv" event={"ID":"bb802929-b2d0-4191-9f35-4cfee6005793","Type":"ContainerDied","Data":"c677b91acc9337b23516ef8080181509878a5abea8ca4c9b8e755dcf26913beb"} Nov 03 00:45:47.082289 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:47.082268 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c677b91acc9337b23516ef8080181509878a5abea8ca4c9b8e755dcf26913beb" Nov 03 00:45:47.083527 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:47.082094 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a31d8e-8dgpv" Nov 03 00:45:53.793363 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:53.793326 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ec1f0a2-b7qmq"] Nov 03 00:45:53.793812 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:53.793682 5114 cpu_manager.go:414] "containerMap: removing container" podUID="bb802929-b2d0-4191-9f35-4cfee6005793" containerName="revoke-keys" Nov 03 00:45:53.793812 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:53.793694 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb802929-b2d0-4191-9f35-4cfee6005793" containerName="revoke-keys" Nov 03 00:45:53.793812 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:53.793756 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb802929-b2d0-4191-9f35-4cfee6005793" containerName="revoke-keys" Nov 03 00:45:53.798155 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:53.798128 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec1f0a2-b7qmq" Nov 03 00:45:53.852457 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:53.852419 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:45:53.901760 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:53.901716 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:45:53.901760 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:53.901747 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:45:53.902059 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:53.901901 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:45:53.903583 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:53.903555 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ec1f0a2-b7qmq"] Nov 03 00:45:53.939704 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:53.939662 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7b5831e-4ed3-42db-8c3e-1236ccd8d122-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ec1f0a2-b7qmq\" (UID: \"b7b5831e-4ed3-42db-8c3e-1236ccd8d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec1f0a2-b7qmq" Nov 03 00:45:53.939924 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:53.939730 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7b5831e-4ed3-42db-8c3e-1236ccd8d122-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ec1f0a2-b7qmq\" (UID: \"b7b5831e-4ed3-42db-8c3e-1236ccd8d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec1f0a2-b7qmq" Nov 03 00:45:53.940003 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:53.939968 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmv9n\" (UniqueName: \"kubernetes.io/projected/b7b5831e-4ed3-42db-8c3e-1236ccd8d122-kube-api-access-vmv9n\") pod \"maas-api-revoke-sub-2ec1f0a2-b7qmq\" (UID: \"b7b5831e-4ed3-42db-8c3e-1236ccd8d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec1f0a2-b7qmq" Nov 03 00:45:54.040695 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:54.040656 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7b5831e-4ed3-42db-8c3e-1236ccd8d122-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ec1f0a2-b7qmq\" (UID: \"b7b5831e-4ed3-42db-8c3e-1236ccd8d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec1f0a2-b7qmq" Nov 03 00:45:54.040926 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:54.040738 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmv9n\" (UniqueName: \"kubernetes.io/projected/b7b5831e-4ed3-42db-8c3e-1236ccd8d122-kube-api-access-vmv9n\") pod \"maas-api-revoke-sub-2ec1f0a2-b7qmq\" (UID: \"b7b5831e-4ed3-42db-8c3e-1236ccd8d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec1f0a2-b7qmq" Nov 03 00:45:54.040926 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:54.040810 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7b5831e-4ed3-42db-8c3e-1236ccd8d122-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ec1f0a2-b7qmq\" (UID: \"b7b5831e-4ed3-42db-8c3e-1236ccd8d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec1f0a2-b7qmq" Nov 03 00:45:54.041368 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:54.041343 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7b5831e-4ed3-42db-8c3e-1236ccd8d122-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ec1f0a2-b7qmq\" (UID: \"b7b5831e-4ed3-42db-8c3e-1236ccd8d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec1f0a2-b7qmq" Nov 03 00:45:54.043542 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:54.043464 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7b5831e-4ed3-42db-8c3e-1236ccd8d122-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ec1f0a2-b7qmq\" (UID: \"b7b5831e-4ed3-42db-8c3e-1236ccd8d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec1f0a2-b7qmq" Nov 03 00:45:54.101931 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:54.101889 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmv9n\" (UniqueName: \"kubernetes.io/projected/b7b5831e-4ed3-42db-8c3e-1236ccd8d122-kube-api-access-vmv9n\") pod \"maas-api-revoke-sub-2ec1f0a2-b7qmq\" (UID: \"b7b5831e-4ed3-42db-8c3e-1236ccd8d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec1f0a2-b7qmq" Nov 03 00:45:54.115801 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:54.115767 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec1f0a2-b7qmq" Nov 03 00:45:54.488561 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:54.488536 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ec1f0a2-b7qmq"] Nov 03 00:45:54.490703 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:45:54.490670 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b5831e_4ed3_42db_8c3e_1236ccd8d122.slice/crio-89c08cccd2bb395f2b7a3d799f450238e2ba0428c15c32263bfdad373f098237 WatchSource:0}: Error finding container 89c08cccd2bb395f2b7a3d799f450238e2ba0428c15c32263bfdad373f098237: Status 404 returned error can't find the container with id 89c08cccd2bb395f2b7a3d799f450238e2ba0428c15c32263bfdad373f098237 Nov 03 00:45:55.119934 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:55.119839 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec1f0a2-b7qmq" event={"ID":"b7b5831e-4ed3-42db-8c3e-1236ccd8d122","Type":"ContainerStarted","Data":"fa20e5626f9c19a0b0d85e65fb2c282d839962967d5aab2fbe4a7d06c100ea10"} Nov 03 00:45:55.119934 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:55.119898 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec1f0a2-b7qmq" event={"ID":"b7b5831e-4ed3-42db-8c3e-1236ccd8d122","Type":"ContainerStarted","Data":"89c08cccd2bb395f2b7a3d799f450238e2ba0428c15c32263bfdad373f098237"} Nov 03 00:45:55.236424 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:55.236362 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec1f0a2-b7qmq" podStartSLOduration=2.236347068 podStartE2EDuration="2.236347068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:55.234782422 +0000 UTC m=+7030.587815798" watchObservedRunningTime="2026-11-03 00:45:55.236347068 +0000 UTC m=+7030.589380443" Nov 03 00:45:56.126401 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:56.126361 5114 generic.go:356] "Generic (PLEG): container finished" podID="b7b5831e-4ed3-42db-8c3e-1236ccd8d122" containerID="fa20e5626f9c19a0b0d85e65fb2c282d839962967d5aab2fbe4a7d06c100ea10" exitCode=0 Nov 03 00:45:56.126851 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:56.126425 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec1f0a2-b7qmq" event={"ID":"b7b5831e-4ed3-42db-8c3e-1236ccd8d122","Type":"ContainerDied","Data":"fa20e5626f9c19a0b0d85e65fb2c282d839962967d5aab2fbe4a7d06c100ea10"} Nov 03 00:45:57.267182 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:57.267152 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec1f0a2-b7qmq" Nov 03 00:45:57.367227 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:57.367182 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7b5831e-4ed3-42db-8c3e-1236ccd8d122-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7b5831e-4ed3-42db-8c3e-1236ccd8d122-maas-api-openshift-ca\") pod \"b7b5831e-4ed3-42db-8c3e-1236ccd8d122\" (UID: \"b7b5831e-4ed3-42db-8c3e-1236ccd8d122\") " Nov 03 00:45:57.367227 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:57.367235 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7b5831e-4ed3-42db-8c3e-1236ccd8d122-kube-api-access-vmv9n\" (UniqueName: \"kubernetes.io/projected/b7b5831e-4ed3-42db-8c3e-1236ccd8d122-kube-api-access-vmv9n\") pod \"b7b5831e-4ed3-42db-8c3e-1236ccd8d122\" (UID: \"b7b5831e-4ed3-42db-8c3e-1236ccd8d122\") " Nov 03 00:45:57.367465 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:57.367271 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7b5831e-4ed3-42db-8c3e-1236ccd8d122-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7b5831e-4ed3-42db-8c3e-1236ccd8d122-maas-api-xks-ca\") pod \"b7b5831e-4ed3-42db-8c3e-1236ccd8d122\" (UID: \"b7b5831e-4ed3-42db-8c3e-1236ccd8d122\") " Nov 03 00:45:57.367617 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:57.367592 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7b5831e-4ed3-42db-8c3e-1236ccd8d122-maas-api-openshift-ca" pod "b7b5831e-4ed3-42db-8c3e-1236ccd8d122" (UID: "b7b5831e-4ed3-42db-8c3e-1236ccd8d122"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:45:57.369743 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:57.369704 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7b5831e-4ed3-42db-8c3e-1236ccd8d122-maas-api-xks-ca" pod "b7b5831e-4ed3-42db-8c3e-1236ccd8d122" (UID: "b7b5831e-4ed3-42db-8c3e-1236ccd8d122"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:45:57.369743 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:57.369728 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7b5831e-4ed3-42db-8c3e-1236ccd8d122-kube-api-access-vmv9n" pod "b7b5831e-4ed3-42db-8c3e-1236ccd8d122" (UID: "b7b5831e-4ed3-42db-8c3e-1236ccd8d122"). InnerVolumeSpecName "kube-api-access-vmv9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:45:57.468534 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:57.468432 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7b5831e-4ed3-42db-8c3e-1236ccd8d122-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:57.468534 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:57.468461 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmv9n\" (UniqueName: \"kubernetes.io/projected/b7b5831e-4ed3-42db-8c3e-1236ccd8d122-kube-api-access-vmv9n\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:57.468534 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:57.468474 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7b5831e-4ed3-42db-8c3e-1236ccd8d122-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:58.138814 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:58.138769 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec1f0a2-b7qmq" event={"ID":"b7b5831e-4ed3-42db-8c3e-1236ccd8d122","Type":"ContainerDied","Data":"89c08cccd2bb395f2b7a3d799f450238e2ba0428c15c32263bfdad373f098237"} Nov 03 00:45:58.139007 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:58.138844 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89c08cccd2bb395f2b7a3d799f450238e2ba0428c15c32263bfdad373f098237" Nov 03 00:45:58.139007 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:45:58.138780 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec1f0a2-b7qmq" Nov 03 00:46:04.539665 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:04.539627 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5a01859-j64l6"] Nov 03 00:46:04.540245 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:04.540126 5114 cpu_manager.go:414] "containerMap: removing container" podUID="b7b5831e-4ed3-42db-8c3e-1236ccd8d122" containerName="revoke-keys" Nov 03 00:46:04.540245 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:04.540147 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7b5831e-4ed3-42db-8c3e-1236ccd8d122" containerName="revoke-keys" Nov 03 00:46:04.540370 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:04.540255 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7b5831e-4ed3-42db-8c3e-1236ccd8d122" containerName="revoke-keys" Nov 03 00:46:04.544936 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:04.544911 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a01859-j64l6" Nov 03 00:46:04.633315 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:04.633281 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983a5673-f7b4-4c32-830a-daa08ffd4ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5a01859-j64l6\" (UID: \"983a5673-f7b4-4c32-830a-daa08ffd4ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a01859-j64l6" Nov 03 00:46:04.633536 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:04.633331 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fg75\" (UniqueName: \"kubernetes.io/projected/983a5673-f7b4-4c32-830a-daa08ffd4ea4-kube-api-access-6fg75\") pod \"maas-api-revoke-sub-e5a01859-j64l6\" (UID: \"983a5673-f7b4-4c32-830a-daa08ffd4ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a01859-j64l6" Nov 03 00:46:04.633536 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:04.633444 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983a5673-f7b4-4c32-830a-daa08ffd4ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5a01859-j64l6\" (UID: \"983a5673-f7b4-4c32-830a-daa08ffd4ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a01859-j64l6" Nov 03 00:46:04.649877 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:04.649837 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:46:04.649877 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:04.649860 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:46:04.650091 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:04.649972 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:46:04.650259 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:04.650241 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:46:04.652055 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:04.652034 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5a01859-j64l6"] Nov 03 00:46:04.733928 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:04.733895 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983a5673-f7b4-4c32-830a-daa08ffd4ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5a01859-j64l6\" (UID: \"983a5673-f7b4-4c32-830a-daa08ffd4ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a01859-j64l6" Nov 03 00:46:04.734123 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:04.734004 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983a5673-f7b4-4c32-830a-daa08ffd4ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5a01859-j64l6\" (UID: \"983a5673-f7b4-4c32-830a-daa08ffd4ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a01859-j64l6" Nov 03 00:46:04.734123 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:04.734068 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fg75\" (UniqueName: \"kubernetes.io/projected/983a5673-f7b4-4c32-830a-daa08ffd4ea4-kube-api-access-6fg75\") pod \"maas-api-revoke-sub-e5a01859-j64l6\" (UID: \"983a5673-f7b4-4c32-830a-daa08ffd4ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a01859-j64l6" Nov 03 00:46:04.734686 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:04.734659 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983a5673-f7b4-4c32-830a-daa08ffd4ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5a01859-j64l6\" (UID: \"983a5673-f7b4-4c32-830a-daa08ffd4ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a01859-j64l6" Nov 03 00:46:04.736621 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:04.736595 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983a5673-f7b4-4c32-830a-daa08ffd4ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5a01859-j64l6\" (UID: \"983a5673-f7b4-4c32-830a-daa08ffd4ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a01859-j64l6" Nov 03 00:46:04.795099 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:04.795024 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fg75\" (UniqueName: \"kubernetes.io/projected/983a5673-f7b4-4c32-830a-daa08ffd4ea4-kube-api-access-6fg75\") pod \"maas-api-revoke-sub-e5a01859-j64l6\" (UID: \"983a5673-f7b4-4c32-830a-daa08ffd4ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a01859-j64l6" Nov 03 00:46:04.857779 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:04.857740 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a01859-j64l6" Nov 03 00:46:05.231182 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:05.231149 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5a01859-j64l6"] Nov 03 00:46:05.233400 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:46:05.233366 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983a5673_f7b4_4c32_830a_daa08ffd4ea4.slice/crio-99e7b068d6d04e0ef61dcf7fc32ba043b3700028677dce3429f28a7f333dcf10 WatchSource:0}: Error finding container 99e7b068d6d04e0ef61dcf7fc32ba043b3700028677dce3429f28a7f333dcf10: Status 404 returned error can't find the container with id 99e7b068d6d04e0ef61dcf7fc32ba043b3700028677dce3429f28a7f333dcf10 Nov 03 00:46:06.219208 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:06.219175 5114 generic.go:356] "Generic (PLEG): container finished" podID="983a5673-f7b4-4c32-830a-daa08ffd4ea4" containerID="08c9eb672eee6d804967d7a918f3859c2f64d593e47d675655b310ba82cd3d6f" exitCode=0 Nov 03 00:46:06.219747 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:06.219225 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a01859-j64l6" event={"ID":"983a5673-f7b4-4c32-830a-daa08ffd4ea4","Type":"ContainerDied","Data":"08c9eb672eee6d804967d7a918f3859c2f64d593e47d675655b310ba82cd3d6f"} Nov 03 00:46:06.219747 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:06.219291 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a01859-j64l6" event={"ID":"983a5673-f7b4-4c32-830a-daa08ffd4ea4","Type":"ContainerStarted","Data":"99e7b068d6d04e0ef61dcf7fc32ba043b3700028677dce3429f28a7f333dcf10"} Nov 03 00:46:07.362209 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:07.362181 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a01859-j64l6" Nov 03 00:46:07.459365 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:07.459331 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/983a5673-f7b4-4c32-830a-daa08ffd4ea4-kube-api-access-6fg75\" (UniqueName: \"kubernetes.io/projected/983a5673-f7b4-4c32-830a-daa08ffd4ea4-kube-api-access-6fg75\") pod \"983a5673-f7b4-4c32-830a-daa08ffd4ea4\" (UID: \"983a5673-f7b4-4c32-830a-daa08ffd4ea4\") " Nov 03 00:46:07.459613 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:07.459406 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/983a5673-f7b4-4c32-830a-daa08ffd4ea4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983a5673-f7b4-4c32-830a-daa08ffd4ea4-maas-api-xks-ca\") pod \"983a5673-f7b4-4c32-830a-daa08ffd4ea4\" (UID: \"983a5673-f7b4-4c32-830a-daa08ffd4ea4\") " Nov 03 00:46:07.459613 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:07.459471 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/983a5673-f7b4-4c32-830a-daa08ffd4ea4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983a5673-f7b4-4c32-830a-daa08ffd4ea4-maas-api-openshift-ca\") pod \"983a5673-f7b4-4c32-830a-daa08ffd4ea4\" (UID: \"983a5673-f7b4-4c32-830a-daa08ffd4ea4\") " Nov 03 00:46:07.459866 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:07.459843 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/983a5673-f7b4-4c32-830a-daa08ffd4ea4-maas-api-openshift-ca" pod "983a5673-f7b4-4c32-830a-daa08ffd4ea4" (UID: "983a5673-f7b4-4c32-830a-daa08ffd4ea4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:46:07.461745 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:07.461719 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/983a5673-f7b4-4c32-830a-daa08ffd4ea4-maas-api-xks-ca" pod "983a5673-f7b4-4c32-830a-daa08ffd4ea4" (UID: "983a5673-f7b4-4c32-830a-daa08ffd4ea4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:46:07.461870 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:07.461850 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983a5673-f7b4-4c32-830a-daa08ffd4ea4-kube-api-access-6fg75" pod "983a5673-f7b4-4c32-830a-daa08ffd4ea4" (UID: "983a5673-f7b4-4c32-830a-daa08ffd4ea4"). InnerVolumeSpecName "kube-api-access-6fg75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:46:07.560261 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:07.560225 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983a5673-f7b4-4c32-830a-daa08ffd4ea4-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:07.560261 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:07.560253 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983a5673-f7b4-4c32-830a-daa08ffd4ea4-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:07.560261 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:07.560265 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fg75\" (UniqueName: \"kubernetes.io/projected/983a5673-f7b4-4c32-830a-daa08ffd4ea4-kube-api-access-6fg75\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:07.582952 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:46:07.582912 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:46:08.230286 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:08.230249 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a01859-j64l6" event={"ID":"983a5673-f7b4-4c32-830a-daa08ffd4ea4","Type":"ContainerDied","Data":"99e7b068d6d04e0ef61dcf7fc32ba043b3700028677dce3429f28a7f333dcf10"} Nov 03 00:46:08.230286 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:08.230270 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a01859-j64l6" Nov 03 00:46:08.230538 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:08.230306 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99e7b068d6d04e0ef61dcf7fc32ba043b3700028677dce3429f28a7f333dcf10" Nov 03 00:46:15.547083 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:15.547042 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e24a2c-6zhhh"] Nov 03 00:46:15.547627 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:15.547378 5114 cpu_manager.go:414] "containerMap: removing container" podUID="983a5673-f7b4-4c32-830a-daa08ffd4ea4" containerName="revoke-keys" Nov 03 00:46:15.547627 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:15.547388 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="983a5673-f7b4-4c32-830a-daa08ffd4ea4" containerName="revoke-keys" Nov 03 00:46:15.547627 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:15.547449 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="983a5673-f7b4-4c32-830a-daa08ffd4ea4" containerName="revoke-keys" Nov 03 00:46:15.550601 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:15.550581 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e24a2c-6zhhh" Nov 03 00:46:15.602799 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:15.602761 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:46:15.602799 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:15.602763 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:46:15.603070 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:15.602833 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:46:15.603189 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:15.603165 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:46:15.608851 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:15.608821 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e24a2c-6zhhh"] Nov 03 00:46:15.636009 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:15.635973 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b48023-8186-4e7a-9cb8-2b4950e796b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8e24a2c-6zhhh\" (UID: \"44b48023-8186-4e7a-9cb8-2b4950e796b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e24a2c-6zhhh" Nov 03 00:46:15.636225 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:15.636119 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmk6f\" (UniqueName: \"kubernetes.io/projected/44b48023-8186-4e7a-9cb8-2b4950e796b7-kube-api-access-dmk6f\") pod \"maas-api-revoke-sub-c8e24a2c-6zhhh\" (UID: \"44b48023-8186-4e7a-9cb8-2b4950e796b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e24a2c-6zhhh" Nov 03 00:46:15.636225 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:15.636154 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b48023-8186-4e7a-9cb8-2b4950e796b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8e24a2c-6zhhh\" (UID: \"44b48023-8186-4e7a-9cb8-2b4950e796b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e24a2c-6zhhh" Nov 03 00:46:15.737157 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:15.737116 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmk6f\" (UniqueName: \"kubernetes.io/projected/44b48023-8186-4e7a-9cb8-2b4950e796b7-kube-api-access-dmk6f\") pod \"maas-api-revoke-sub-c8e24a2c-6zhhh\" (UID: \"44b48023-8186-4e7a-9cb8-2b4950e796b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e24a2c-6zhhh" Nov 03 00:46:15.737369 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:15.737163 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b48023-8186-4e7a-9cb8-2b4950e796b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8e24a2c-6zhhh\" (UID: \"44b48023-8186-4e7a-9cb8-2b4950e796b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e24a2c-6zhhh" Nov 03 00:46:15.737369 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:15.737214 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b48023-8186-4e7a-9cb8-2b4950e796b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8e24a2c-6zhhh\" (UID: \"44b48023-8186-4e7a-9cb8-2b4950e796b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e24a2c-6zhhh" Nov 03 00:46:15.737951 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:15.737922 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b48023-8186-4e7a-9cb8-2b4950e796b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8e24a2c-6zhhh\" (UID: \"44b48023-8186-4e7a-9cb8-2b4950e796b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e24a2c-6zhhh" Nov 03 00:46:15.739978 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:15.739944 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b48023-8186-4e7a-9cb8-2b4950e796b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8e24a2c-6zhhh\" (UID: \"44b48023-8186-4e7a-9cb8-2b4950e796b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e24a2c-6zhhh" Nov 03 00:46:15.798429 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:15.798348 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmk6f\" (UniqueName: \"kubernetes.io/projected/44b48023-8186-4e7a-9cb8-2b4950e796b7-kube-api-access-dmk6f\") pod \"maas-api-revoke-sub-c8e24a2c-6zhhh\" (UID: \"44b48023-8186-4e7a-9cb8-2b4950e796b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e24a2c-6zhhh" Nov 03 00:46:15.863511 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:15.863451 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e24a2c-6zhhh" Nov 03 00:46:16.238423 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:16.238398 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e24a2c-6zhhh"] Nov 03 00:46:16.240606 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:46:16.240571 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b48023_8186_4e7a_9cb8_2b4950e796b7.slice/crio-f7aafa464d6ecb4abbe07d7974604e3f67fb0947bb213648567d56bccb876745 WatchSource:0}: Error finding container f7aafa464d6ecb4abbe07d7974604e3f67fb0947bb213648567d56bccb876745: Status 404 returned error can't find the container with id f7aafa464d6ecb4abbe07d7974604e3f67fb0947bb213648567d56bccb876745 Nov 03 00:46:16.268107 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:16.268074 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e24a2c-6zhhh" event={"ID":"44b48023-8186-4e7a-9cb8-2b4950e796b7","Type":"ContainerStarted","Data":"f7aafa464d6ecb4abbe07d7974604e3f67fb0947bb213648567d56bccb876745"} Nov 03 00:46:17.274813 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:17.274773 5114 generic.go:356] "Generic (PLEG): container finished" podID="44b48023-8186-4e7a-9cb8-2b4950e796b7" containerID="51fcb16200c5040c4edeb1d9ee390c0dceb6e0bb06854e07627c23462e100808" exitCode=0 Nov 03 00:46:17.274813 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:17.274816 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e24a2c-6zhhh" event={"ID":"44b48023-8186-4e7a-9cb8-2b4950e796b7","Type":"ContainerDied","Data":"51fcb16200c5040c4edeb1d9ee390c0dceb6e0bb06854e07627c23462e100808"} Nov 03 00:46:18.447414 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:18.447375 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e24a2c-6zhhh" Nov 03 00:46:18.564669 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:18.564632 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44b48023-8186-4e7a-9cb8-2b4950e796b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b48023-8186-4e7a-9cb8-2b4950e796b7-maas-api-xks-ca\") pod \"44b48023-8186-4e7a-9cb8-2b4950e796b7\" (UID: \"44b48023-8186-4e7a-9cb8-2b4950e796b7\") " Nov 03 00:46:18.564942 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:18.564756 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b48023-8186-4e7a-9cb8-2b4950e796b7-kube-api-access-dmk6f\" (UniqueName: \"kubernetes.io/projected/44b48023-8186-4e7a-9cb8-2b4950e796b7-kube-api-access-dmk6f\") pod \"44b48023-8186-4e7a-9cb8-2b4950e796b7\" (UID: \"44b48023-8186-4e7a-9cb8-2b4950e796b7\") " Nov 03 00:46:18.564942 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:18.564853 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44b48023-8186-4e7a-9cb8-2b4950e796b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b48023-8186-4e7a-9cb8-2b4950e796b7-maas-api-openshift-ca\") pod \"44b48023-8186-4e7a-9cb8-2b4950e796b7\" (UID: \"44b48023-8186-4e7a-9cb8-2b4950e796b7\") " Nov 03 00:46:18.565891 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:18.565849 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b48023-8186-4e7a-9cb8-2b4950e796b7-maas-api-openshift-ca" pod "44b48023-8186-4e7a-9cb8-2b4950e796b7" (UID: "44b48023-8186-4e7a-9cb8-2b4950e796b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:46:18.567694 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:18.567622 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b48023-8186-4e7a-9cb8-2b4950e796b7-kube-api-access-dmk6f" pod "44b48023-8186-4e7a-9cb8-2b4950e796b7" (UID: "44b48023-8186-4e7a-9cb8-2b4950e796b7"). InnerVolumeSpecName "kube-api-access-dmk6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:46:18.568551 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:18.568364 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b48023-8186-4e7a-9cb8-2b4950e796b7-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:18.568551 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:18.568397 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmk6f\" (UniqueName: \"kubernetes.io/projected/44b48023-8186-4e7a-9cb8-2b4950e796b7-kube-api-access-dmk6f\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:18.570235 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:18.570199 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44b48023-8186-4e7a-9cb8-2b4950e796b7-maas-api-xks-ca" pod "44b48023-8186-4e7a-9cb8-2b4950e796b7" (UID: "44b48023-8186-4e7a-9cb8-2b4950e796b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:46:18.669693 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:18.669654 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b48023-8186-4e7a-9cb8-2b4950e796b7-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:19.288811 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:19.288782 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e24a2c-6zhhh" Nov 03 00:46:19.289036 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:19.288827 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e24a2c-6zhhh" event={"ID":"44b48023-8186-4e7a-9cb8-2b4950e796b7","Type":"ContainerDied","Data":"f7aafa464d6ecb4abbe07d7974604e3f67fb0947bb213648567d56bccb876745"} Nov 03 00:46:19.289036 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:19.288889 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7aafa464d6ecb4abbe07d7974604e3f67fb0947bb213648567d56bccb876745" Nov 03 00:46:26.450371 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:26.450326 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d0d532c-6bxv7"] Nov 03 00:46:26.450817 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:26.450777 5114 cpu_manager.go:414] "containerMap: removing container" podUID="44b48023-8186-4e7a-9cb8-2b4950e796b7" containerName="revoke-keys" Nov 03 00:46:26.450817 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:26.450794 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b48023-8186-4e7a-9cb8-2b4950e796b7" containerName="revoke-keys" Nov 03 00:46:26.450893 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:26.450867 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b48023-8186-4e7a-9cb8-2b4950e796b7" containerName="revoke-keys" Nov 03 00:46:26.453851 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:26.453828 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0d532c-6bxv7" Nov 03 00:46:26.511849 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:26.511808 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d0d532c-6bxv7"] Nov 03 00:46:26.548803 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:26.548764 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07397238-7175-458f-9965-087261e39f88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d0d532c-6bxv7\" (UID: \"07397238-7175-458f-9965-087261e39f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0d532c-6bxv7" Nov 03 00:46:26.549033 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:26.548887 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6d5m\" (UniqueName: \"kubernetes.io/projected/07397238-7175-458f-9965-087261e39f88-kube-api-access-h6d5m\") pod \"maas-api-revoke-sub-6d0d532c-6bxv7\" (UID: \"07397238-7175-458f-9965-087261e39f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0d532c-6bxv7" Nov 03 00:46:26.549033 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:26.548986 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07397238-7175-458f-9965-087261e39f88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d0d532c-6bxv7\" (UID: \"07397238-7175-458f-9965-087261e39f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0d532c-6bxv7" Nov 03 00:46:26.557959 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:26.557915 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:46:26.557959 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:26.557919 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:46:26.557959 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:26.557936 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:46:26.558177 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:26.558000 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:46:26.650531 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:26.650463 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07397238-7175-458f-9965-087261e39f88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d0d532c-6bxv7\" (UID: \"07397238-7175-458f-9965-087261e39f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0d532c-6bxv7" Nov 03 00:46:26.650531 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:26.650529 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07397238-7175-458f-9965-087261e39f88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d0d532c-6bxv7\" (UID: \"07397238-7175-458f-9965-087261e39f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0d532c-6bxv7" Nov 03 00:46:26.650767 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:26.650574 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6d5m\" (UniqueName: \"kubernetes.io/projected/07397238-7175-458f-9965-087261e39f88-kube-api-access-h6d5m\") pod \"maas-api-revoke-sub-6d0d532c-6bxv7\" (UID: \"07397238-7175-458f-9965-087261e39f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0d532c-6bxv7" Nov 03 00:46:26.651201 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:26.651179 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07397238-7175-458f-9965-087261e39f88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d0d532c-6bxv7\" (UID: \"07397238-7175-458f-9965-087261e39f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0d532c-6bxv7" Nov 03 00:46:26.653189 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:26.653167 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07397238-7175-458f-9965-087261e39f88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d0d532c-6bxv7\" (UID: \"07397238-7175-458f-9965-087261e39f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0d532c-6bxv7" Nov 03 00:46:26.711342 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:26.711262 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6d5m\" (UniqueName: \"kubernetes.io/projected/07397238-7175-458f-9965-087261e39f88-kube-api-access-h6d5m\") pod \"maas-api-revoke-sub-6d0d532c-6bxv7\" (UID: \"07397238-7175-458f-9965-087261e39f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0d532c-6bxv7" Nov 03 00:46:26.766691 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:26.766649 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0d532c-6bxv7" Nov 03 00:46:27.140873 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:27.140820 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d0d532c-6bxv7"] Nov 03 00:46:27.143588 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:46:27.143555 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07397238_7175_458f_9965_087261e39f88.slice/crio-350f61fd6a279e358b7a72e4186ab4452773b2c706d271091ca9f4929d9a2d2b WatchSource:0}: Error finding container 350f61fd6a279e358b7a72e4186ab4452773b2c706d271091ca9f4929d9a2d2b: Status 404 returned error can't find the container with id 350f61fd6a279e358b7a72e4186ab4452773b2c706d271091ca9f4929d9a2d2b Nov 03 00:46:27.328874 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:27.328840 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0d532c-6bxv7" event={"ID":"07397238-7175-458f-9965-087261e39f88","Type":"ContainerStarted","Data":"350f61fd6a279e358b7a72e4186ab4452773b2c706d271091ca9f4929d9a2d2b"} Nov 03 00:46:28.335034 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:28.334987 5114 generic.go:356] "Generic (PLEG): container finished" podID="07397238-7175-458f-9965-087261e39f88" containerID="7c633d40740c7881b480a3c3f447ab8369d8135f585c7603e14545c2d218be2c" exitCode=0 Nov 03 00:46:28.335552 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:28.335067 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0d532c-6bxv7" event={"ID":"07397238-7175-458f-9965-087261e39f88","Type":"ContainerDied","Data":"7c633d40740c7881b480a3c3f447ab8369d8135f585c7603e14545c2d218be2c"} Nov 03 00:46:29.507566 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:29.507525 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0d532c-6bxv7" Nov 03 00:46:29.574104 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:29.574058 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07397238-7175-458f-9965-087261e39f88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07397238-7175-458f-9965-087261e39f88-maas-api-openshift-ca\") pod \"07397238-7175-458f-9965-087261e39f88\" (UID: \"07397238-7175-458f-9965-087261e39f88\") " Nov 03 00:46:29.574333 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:29.574125 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07397238-7175-458f-9965-087261e39f88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07397238-7175-458f-9965-087261e39f88-maas-api-xks-ca\") pod \"07397238-7175-458f-9965-087261e39f88\" (UID: \"07397238-7175-458f-9965-087261e39f88\") " Nov 03 00:46:29.574333 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:29.574198 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07397238-7175-458f-9965-087261e39f88-kube-api-access-h6d5m\" (UniqueName: \"kubernetes.io/projected/07397238-7175-458f-9965-087261e39f88-kube-api-access-h6d5m\") pod \"07397238-7175-458f-9965-087261e39f88\" (UID: \"07397238-7175-458f-9965-087261e39f88\") " Nov 03 00:46:29.574603 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:29.574573 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07397238-7175-458f-9965-087261e39f88-maas-api-openshift-ca" pod "07397238-7175-458f-9965-087261e39f88" (UID: "07397238-7175-458f-9965-087261e39f88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:46:29.577162 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:29.577132 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07397238-7175-458f-9965-087261e39f88-maas-api-xks-ca" pod "07397238-7175-458f-9965-087261e39f88" (UID: "07397238-7175-458f-9965-087261e39f88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:46:29.577431 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:29.577399 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07397238-7175-458f-9965-087261e39f88-kube-api-access-h6d5m" pod "07397238-7175-458f-9965-087261e39f88" (UID: "07397238-7175-458f-9965-087261e39f88"). InnerVolumeSpecName "kube-api-access-h6d5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:46:29.675855 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:29.675760 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07397238-7175-458f-9965-087261e39f88-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:29.675855 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:29.675810 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6d5m\" (UniqueName: \"kubernetes.io/projected/07397238-7175-458f-9965-087261e39f88-kube-api-access-h6d5m\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:29.675855 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:29.675828 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07397238-7175-458f-9965-087261e39f88-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:30.373513 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:30.369896 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0d532c-6bxv7" event={"ID":"07397238-7175-458f-9965-087261e39f88","Type":"ContainerDied","Data":"350f61fd6a279e358b7a72e4186ab4452773b2c706d271091ca9f4929d9a2d2b"} Nov 03 00:46:30.373513 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:30.369970 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="350f61fd6a279e358b7a72e4186ab4452773b2c706d271091ca9f4929d9a2d2b" Nov 03 00:46:30.373513 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:30.370050 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0d532c-6bxv7" Nov 03 00:46:37.199648 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:37.199608 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c4e171f-nlvvf"] Nov 03 00:46:37.200148 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:37.199970 5114 cpu_manager.go:414] "containerMap: removing container" podUID="07397238-7175-458f-9965-087261e39f88" containerName="revoke-keys" Nov 03 00:46:37.200148 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:37.199988 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07397238-7175-458f-9965-087261e39f88" containerName="revoke-keys" Nov 03 00:46:37.200148 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:37.200064 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="07397238-7175-458f-9965-087261e39f88" containerName="revoke-keys" Nov 03 00:46:37.207831 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:37.207806 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c4e171f-nlvvf" Nov 03 00:46:37.245520 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:37.245457 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a1343f-c07d-4c54-9fdf-28f2e20c4caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c4e171f-nlvvf\" (UID: \"64a1343f-c07d-4c54-9fdf-28f2e20c4caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c4e171f-nlvvf" Nov 03 00:46:37.245700 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:37.245659 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a1343f-c07d-4c54-9fdf-28f2e20c4caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c4e171f-nlvvf\" (UID: \"64a1343f-c07d-4c54-9fdf-28f2e20c4caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c4e171f-nlvvf" Nov 03 00:46:37.245700 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:37.245683 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqqh7\" (UniqueName: \"kubernetes.io/projected/64a1343f-c07d-4c54-9fdf-28f2e20c4caf-kube-api-access-tqqh7\") pod \"maas-api-revoke-sub-1c4e171f-nlvvf\" (UID: \"64a1343f-c07d-4c54-9fdf-28f2e20c4caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c4e171f-nlvvf" Nov 03 00:46:37.260252 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:37.260201 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:46:37.260252 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:37.260217 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:46:37.261080 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:37.261056 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c4e171f-nlvvf"] Nov 03 00:46:37.308222 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:37.308182 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:46:37.308222 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:37.308193 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:46:37.346741 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:37.346706 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqqh7\" (UniqueName: \"kubernetes.io/projected/64a1343f-c07d-4c54-9fdf-28f2e20c4caf-kube-api-access-tqqh7\") pod \"maas-api-revoke-sub-1c4e171f-nlvvf\" (UID: \"64a1343f-c07d-4c54-9fdf-28f2e20c4caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c4e171f-nlvvf" Nov 03 00:46:37.346924 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:37.346748 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a1343f-c07d-4c54-9fdf-28f2e20c4caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c4e171f-nlvvf\" (UID: \"64a1343f-c07d-4c54-9fdf-28f2e20c4caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c4e171f-nlvvf" Nov 03 00:46:37.346924 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:37.346834 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a1343f-c07d-4c54-9fdf-28f2e20c4caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c4e171f-nlvvf\" (UID: \"64a1343f-c07d-4c54-9fdf-28f2e20c4caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c4e171f-nlvvf" Nov 03 00:46:37.347455 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:37.347427 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a1343f-c07d-4c54-9fdf-28f2e20c4caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c4e171f-nlvvf\" (UID: \"64a1343f-c07d-4c54-9fdf-28f2e20c4caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c4e171f-nlvvf" Nov 03 00:46:37.349444 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:37.349420 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a1343f-c07d-4c54-9fdf-28f2e20c4caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c4e171f-nlvvf\" (UID: \"64a1343f-c07d-4c54-9fdf-28f2e20c4caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c4e171f-nlvvf" Nov 03 00:46:37.408194 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:37.408143 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqqh7\" (UniqueName: \"kubernetes.io/projected/64a1343f-c07d-4c54-9fdf-28f2e20c4caf-kube-api-access-tqqh7\") pod \"maas-api-revoke-sub-1c4e171f-nlvvf\" (UID: \"64a1343f-c07d-4c54-9fdf-28f2e20c4caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c4e171f-nlvvf" Nov 03 00:46:37.521533 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:37.521413 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c4e171f-nlvvf" Nov 03 00:46:37.593141 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:46:37.593097 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:46:37.897123 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:37.897097 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c4e171f-nlvvf"] Nov 03 00:46:37.899380 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:46:37.899342 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a1343f_c07d_4c54_9fdf_28f2e20c4caf.slice/crio-db7b778832662ec268ce9d831bd62633550be4ed24a378a4839d087d3c3dd13b WatchSource:0}: Error finding container db7b778832662ec268ce9d831bd62633550be4ed24a378a4839d087d3c3dd13b: Status 404 returned error can't find the container with id db7b778832662ec268ce9d831bd62633550be4ed24a378a4839d087d3c3dd13b Nov 03 00:46:38.416084 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:38.416043 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c4e171f-nlvvf" event={"ID":"64a1343f-c07d-4c54-9fdf-28f2e20c4caf","Type":"ContainerStarted","Data":"53229fc3bc84bc87f8daf4e30b0c4623a8d1ba530192fa75d10ed0f5b6c4a711"} Nov 03 00:46:38.416639 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:38.416104 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c4e171f-nlvvf" event={"ID":"64a1343f-c07d-4c54-9fdf-28f2e20c4caf","Type":"ContainerStarted","Data":"db7b778832662ec268ce9d831bd62633550be4ed24a378a4839d087d3c3dd13b"} Nov 03 00:46:38.530917 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:38.530798 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c4e171f-nlvvf" podStartSLOduration=1.530781112 podStartE2EDuration="1.530781112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:38.530127364 +0000 UTC m=+7073.883160739" watchObservedRunningTime="2026-11-03 00:46:38.530781112 +0000 UTC m=+7073.883814494" Nov 03 00:46:39.422650 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:39.422616 5114 generic.go:356] "Generic (PLEG): container finished" podID="64a1343f-c07d-4c54-9fdf-28f2e20c4caf" containerID="53229fc3bc84bc87f8daf4e30b0c4623a8d1ba530192fa75d10ed0f5b6c4a711" exitCode=0 Nov 03 00:46:39.423067 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:39.422708 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c4e171f-nlvvf" event={"ID":"64a1343f-c07d-4c54-9fdf-28f2e20c4caf","Type":"ContainerDied","Data":"53229fc3bc84bc87f8daf4e30b0c4623a8d1ba530192fa75d10ed0f5b6c4a711"} Nov 03 00:46:40.603594 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:40.603554 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c4e171f-nlvvf" Nov 03 00:46:40.676044 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:40.675987 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64a1343f-c07d-4c54-9fdf-28f2e20c4caf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a1343f-c07d-4c54-9fdf-28f2e20c4caf-maas-api-openshift-ca\") pod \"64a1343f-c07d-4c54-9fdf-28f2e20c4caf\" (UID: \"64a1343f-c07d-4c54-9fdf-28f2e20c4caf\") " Nov 03 00:46:40.676269 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:40.676105 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64a1343f-c07d-4c54-9fdf-28f2e20c4caf-kube-api-access-tqqh7\" (UniqueName: \"kubernetes.io/projected/64a1343f-c07d-4c54-9fdf-28f2e20c4caf-kube-api-access-tqqh7\") pod \"64a1343f-c07d-4c54-9fdf-28f2e20c4caf\" (UID: \"64a1343f-c07d-4c54-9fdf-28f2e20c4caf\") " Nov 03 00:46:40.676269 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:40.676177 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64a1343f-c07d-4c54-9fdf-28f2e20c4caf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a1343f-c07d-4c54-9fdf-28f2e20c4caf-maas-api-xks-ca\") pod \"64a1343f-c07d-4c54-9fdf-28f2e20c4caf\" (UID: \"64a1343f-c07d-4c54-9fdf-28f2e20c4caf\") " Nov 03 00:46:40.676533 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:40.676472 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a1343f-c07d-4c54-9fdf-28f2e20c4caf-maas-api-openshift-ca" pod "64a1343f-c07d-4c54-9fdf-28f2e20c4caf" (UID: "64a1343f-c07d-4c54-9fdf-28f2e20c4caf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:46:40.678763 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:40.678729 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64a1343f-c07d-4c54-9fdf-28f2e20c4caf-maas-api-xks-ca" pod "64a1343f-c07d-4c54-9fdf-28f2e20c4caf" (UID: "64a1343f-c07d-4c54-9fdf-28f2e20c4caf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:46:40.679017 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:40.678993 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a1343f-c07d-4c54-9fdf-28f2e20c4caf-kube-api-access-tqqh7" pod "64a1343f-c07d-4c54-9fdf-28f2e20c4caf" (UID: "64a1343f-c07d-4c54-9fdf-28f2e20c4caf"). InnerVolumeSpecName "kube-api-access-tqqh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:46:40.779387 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:40.779232 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqqh7\" (UniqueName: \"kubernetes.io/projected/64a1343f-c07d-4c54-9fdf-28f2e20c4caf-kube-api-access-tqqh7\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:40.779387 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:40.779284 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a1343f-c07d-4c54-9fdf-28f2e20c4caf-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:40.779387 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:40.779301 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a1343f-c07d-4c54-9fdf-28f2e20c4caf-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:41.479510 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:41.476591 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c4e171f-nlvvf" Nov 03 00:46:41.479510 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:41.476728 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c4e171f-nlvvf" event={"ID":"64a1343f-c07d-4c54-9fdf-28f2e20c4caf","Type":"ContainerDied","Data":"db7b778832662ec268ce9d831bd62633550be4ed24a378a4839d087d3c3dd13b"} Nov 03 00:46:41.479510 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:41.476822 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db7b778832662ec268ce9d831bd62633550be4ed24a378a4839d087d3c3dd13b" Nov 03 00:46:47.999318 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:47.999274 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16c84053-tp5l9"] Nov 03 00:46:47.999858 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:47.999644 5114 cpu_manager.go:414] "containerMap: removing container" podUID="64a1343f-c07d-4c54-9fdf-28f2e20c4caf" containerName="revoke-keys" Nov 03 00:46:47.999858 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:47.999656 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a1343f-c07d-4c54-9fdf-28f2e20c4caf" containerName="revoke-keys" Nov 03 00:46:47.999858 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:47.999723 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a1343f-c07d-4c54-9fdf-28f2e20c4caf" containerName="revoke-keys" Nov 03 00:46:48.002934 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:48.002912 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c84053-tp5l9" Nov 03 00:46:48.045162 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:48.045118 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt7wz\" (UniqueName: \"kubernetes.io/projected/ed01776b-287a-4d6a-8897-9b65df4b9cfe-kube-api-access-gt7wz\") pod \"maas-api-revoke-sub-16c84053-tp5l9\" (UID: \"ed01776b-287a-4d6a-8897-9b65df4b9cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c84053-tp5l9" Nov 03 00:46:48.045162 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:48.045167 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed01776b-287a-4d6a-8897-9b65df4b9cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16c84053-tp5l9\" (UID: \"ed01776b-287a-4d6a-8897-9b65df4b9cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c84053-tp5l9" Nov 03 00:46:48.045394 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:48.045278 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed01776b-287a-4d6a-8897-9b65df4b9cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16c84053-tp5l9\" (UID: \"ed01776b-287a-4d6a-8897-9b65df4b9cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c84053-tp5l9" Nov 03 00:46:48.107748 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:48.107705 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:46:48.107748 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:48.107742 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:46:48.108034 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:48.107750 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:46:48.108034 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:48.107752 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:46:48.109924 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:48.109901 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16c84053-tp5l9"] Nov 03 00:46:48.146492 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:48.146456 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed01776b-287a-4d6a-8897-9b65df4b9cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16c84053-tp5l9\" (UID: \"ed01776b-287a-4d6a-8897-9b65df4b9cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c84053-tp5l9" Nov 03 00:46:48.146654 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:48.146564 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt7wz\" (UniqueName: \"kubernetes.io/projected/ed01776b-287a-4d6a-8897-9b65df4b9cfe-kube-api-access-gt7wz\") pod \"maas-api-revoke-sub-16c84053-tp5l9\" (UID: \"ed01776b-287a-4d6a-8897-9b65df4b9cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c84053-tp5l9" Nov 03 00:46:48.146654 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:48.146583 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed01776b-287a-4d6a-8897-9b65df4b9cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16c84053-tp5l9\" (UID: \"ed01776b-287a-4d6a-8897-9b65df4b9cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c84053-tp5l9" Nov 03 00:46:48.147200 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:48.147170 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed01776b-287a-4d6a-8897-9b65df4b9cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16c84053-tp5l9\" (UID: \"ed01776b-287a-4d6a-8897-9b65df4b9cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c84053-tp5l9" Nov 03 00:46:48.149134 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:48.149112 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed01776b-287a-4d6a-8897-9b65df4b9cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16c84053-tp5l9\" (UID: \"ed01776b-287a-4d6a-8897-9b65df4b9cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c84053-tp5l9" Nov 03 00:46:48.207623 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:48.207582 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt7wz\" (UniqueName: \"kubernetes.io/projected/ed01776b-287a-4d6a-8897-9b65df4b9cfe-kube-api-access-gt7wz\") pod \"maas-api-revoke-sub-16c84053-tp5l9\" (UID: \"ed01776b-287a-4d6a-8897-9b65df4b9cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c84053-tp5l9" Nov 03 00:46:48.315926 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:48.315886 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c84053-tp5l9" Nov 03 00:46:48.689037 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:48.688934 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16c84053-tp5l9"] Nov 03 00:46:48.691938 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:46:48.691900 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded01776b_287a_4d6a_8897_9b65df4b9cfe.slice/crio-8dbcbd5a15f0e56cafaa80cc9d4db9c06ccac3efb325876ad4b7063d452c7dd0 WatchSource:0}: Error finding container 8dbcbd5a15f0e56cafaa80cc9d4db9c06ccac3efb325876ad4b7063d452c7dd0: Status 404 returned error can't find the container with id 8dbcbd5a15f0e56cafaa80cc9d4db9c06ccac3efb325876ad4b7063d452c7dd0 Nov 03 00:46:49.525346 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:49.525305 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c84053-tp5l9" event={"ID":"ed01776b-287a-4d6a-8897-9b65df4b9cfe","Type":"ContainerStarted","Data":"3c6b27be824902c2dd9b44c21fe5184c8415db7953a0bef46c202cb42c39573c"} Nov 03 00:46:49.525890 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:49.525364 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c84053-tp5l9" event={"ID":"ed01776b-287a-4d6a-8897-9b65df4b9cfe","Type":"ContainerStarted","Data":"8dbcbd5a15f0e56cafaa80cc9d4db9c06ccac3efb325876ad4b7063d452c7dd0"} Nov 03 00:46:49.641704 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:49.641645 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c84053-tp5l9" podStartSLOduration=2.641628085 podStartE2EDuration="2.641628085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:49.639558437 +0000 UTC m=+7084.992591812" watchObservedRunningTime="2026-11-03 00:46:49.641628085 +0000 UTC m=+7084.994661457" Nov 03 00:46:50.532872 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:50.532835 5114 generic.go:356] "Generic (PLEG): container finished" podID="ed01776b-287a-4d6a-8897-9b65df4b9cfe" containerID="3c6b27be824902c2dd9b44c21fe5184c8415db7953a0bef46c202cb42c39573c" exitCode=0 Nov 03 00:46:50.532872 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:50.532882 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c84053-tp5l9" event={"ID":"ed01776b-287a-4d6a-8897-9b65df4b9cfe","Type":"ContainerDied","Data":"3c6b27be824902c2dd9b44c21fe5184c8415db7953a0bef46c202cb42c39573c"} Nov 03 00:46:51.966511 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:51.961954 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c84053-tp5l9" Nov 03 00:46:51.985859 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:51.985742 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed01776b-287a-4d6a-8897-9b65df4b9cfe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed01776b-287a-4d6a-8897-9b65df4b9cfe-maas-api-xks-ca\") pod \"ed01776b-287a-4d6a-8897-9b65df4b9cfe\" (UID: \"ed01776b-287a-4d6a-8897-9b65df4b9cfe\") " Nov 03 00:46:51.988296 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:51.988263 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed01776b-287a-4d6a-8897-9b65df4b9cfe-maas-api-xks-ca" pod "ed01776b-287a-4d6a-8897-9b65df4b9cfe" (UID: "ed01776b-287a-4d6a-8897-9b65df4b9cfe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:46:52.086424 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:52.086386 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed01776b-287a-4d6a-8897-9b65df4b9cfe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed01776b-287a-4d6a-8897-9b65df4b9cfe-maas-api-openshift-ca\") pod \"ed01776b-287a-4d6a-8897-9b65df4b9cfe\" (UID: \"ed01776b-287a-4d6a-8897-9b65df4b9cfe\") " Nov 03 00:46:52.086666 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:52.086437 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed01776b-287a-4d6a-8897-9b65df4b9cfe-kube-api-access-gt7wz\" (UniqueName: \"kubernetes.io/projected/ed01776b-287a-4d6a-8897-9b65df4b9cfe-kube-api-access-gt7wz\") pod \"ed01776b-287a-4d6a-8897-9b65df4b9cfe\" (UID: \"ed01776b-287a-4d6a-8897-9b65df4b9cfe\") " Nov 03 00:46:52.086666 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:52.086630 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed01776b-287a-4d6a-8897-9b65df4b9cfe-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:52.086805 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:52.086771 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed01776b-287a-4d6a-8897-9b65df4b9cfe-maas-api-openshift-ca" pod "ed01776b-287a-4d6a-8897-9b65df4b9cfe" (UID: "ed01776b-287a-4d6a-8897-9b65df4b9cfe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:46:52.088963 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:52.088925 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed01776b-287a-4d6a-8897-9b65df4b9cfe-kube-api-access-gt7wz" pod "ed01776b-287a-4d6a-8897-9b65df4b9cfe" (UID: "ed01776b-287a-4d6a-8897-9b65df4b9cfe"). InnerVolumeSpecName "kube-api-access-gt7wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:46:52.187121 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:52.187022 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed01776b-287a-4d6a-8897-9b65df4b9cfe-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:52.187121 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:52.187058 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt7wz\" (UniqueName: \"kubernetes.io/projected/ed01776b-287a-4d6a-8897-9b65df4b9cfe-kube-api-access-gt7wz\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:52.582406 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:52.582361 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c84053-tp5l9" Nov 03 00:46:52.583313 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:52.582370 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c84053-tp5l9" event={"ID":"ed01776b-287a-4d6a-8897-9b65df4b9cfe","Type":"ContainerDied","Data":"8dbcbd5a15f0e56cafaa80cc9d4db9c06ccac3efb325876ad4b7063d452c7dd0"} Nov 03 00:46:52.583443 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:52.583377 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dbcbd5a15f0e56cafaa80cc9d4db9c06ccac3efb325876ad4b7063d452c7dd0" Nov 03 00:46:52.814510 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:52.811617 5114 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ksnhk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:46:52.814510 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:52.811711 5114 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ksnhk" podUID="713030b6-6b1c-4cf9-b01a-58171f6ad0c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:46:55.054395 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:55.054345 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d056df3-xq7sd"] Nov 03 00:46:55.103992 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:55.103959 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d056df3-xq7sd"] Nov 03 00:46:56.791312 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:56.791277 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="754056bc-5d48-4e08-b6f0-e967ad2787ae" path="/var/lib/kubelet/pods/754056bc-5d48-4e08-b6f0-e967ad2787ae/volumes" Nov 03 00:46:58.768229 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:58.768192 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e7ef370-wbgt7"] Nov 03 00:46:58.768748 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:58.768728 5114 cpu_manager.go:414] "containerMap: removing container" podUID="ed01776b-287a-4d6a-8897-9b65df4b9cfe" containerName="revoke-keys" Nov 03 00:46:58.768792 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:58.768752 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed01776b-287a-4d6a-8897-9b65df4b9cfe" containerName="revoke-keys" Nov 03 00:46:58.768883 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:58.768850 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed01776b-287a-4d6a-8897-9b65df4b9cfe" containerName="revoke-keys" Nov 03 00:46:58.772245 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:58.772219 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7ef370-wbgt7" Nov 03 00:46:58.826965 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:58.826904 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:46:58.852216 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:58.852159 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd39ad63-ddfa-42ef-8c17-c35cef29961e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e7ef370-wbgt7\" (UID: \"bd39ad63-ddfa-42ef-8c17-c35cef29961e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7ef370-wbgt7" Nov 03 00:46:58.852424 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:58.852339 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd39ad63-ddfa-42ef-8c17-c35cef29961e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e7ef370-wbgt7\" (UID: \"bd39ad63-ddfa-42ef-8c17-c35cef29961e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7ef370-wbgt7" Nov 03 00:46:58.852424 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:58.852380 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs47v\" (UniqueName: \"kubernetes.io/projected/bd39ad63-ddfa-42ef-8c17-c35cef29961e-kube-api-access-bs47v\") pod \"maas-api-revoke-sub-1e7ef370-wbgt7\" (UID: \"bd39ad63-ddfa-42ef-8c17-c35cef29961e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7ef370-wbgt7" Nov 03 00:46:58.876689 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:58.876648 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:46:58.876689 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:58.876647 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:46:58.876946 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:58.876769 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:46:58.878806 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:58.878779 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e7ef370-wbgt7"] Nov 03 00:46:58.953192 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:58.953151 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd39ad63-ddfa-42ef-8c17-c35cef29961e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e7ef370-wbgt7\" (UID: \"bd39ad63-ddfa-42ef-8c17-c35cef29961e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7ef370-wbgt7" Nov 03 00:46:58.953192 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:58.953193 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs47v\" (UniqueName: \"kubernetes.io/projected/bd39ad63-ddfa-42ef-8c17-c35cef29961e-kube-api-access-bs47v\") pod \"maas-api-revoke-sub-1e7ef370-wbgt7\" (UID: \"bd39ad63-ddfa-42ef-8c17-c35cef29961e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7ef370-wbgt7" Nov 03 00:46:58.953516 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:58.953251 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd39ad63-ddfa-42ef-8c17-c35cef29961e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e7ef370-wbgt7\" (UID: \"bd39ad63-ddfa-42ef-8c17-c35cef29961e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7ef370-wbgt7" Nov 03 00:46:58.953923 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:58.953901 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd39ad63-ddfa-42ef-8c17-c35cef29961e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e7ef370-wbgt7\" (UID: \"bd39ad63-ddfa-42ef-8c17-c35cef29961e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7ef370-wbgt7" Nov 03 00:46:58.955867 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:58.955843 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd39ad63-ddfa-42ef-8c17-c35cef29961e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e7ef370-wbgt7\" (UID: \"bd39ad63-ddfa-42ef-8c17-c35cef29961e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7ef370-wbgt7" Nov 03 00:46:59.014240 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:59.014199 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs47v\" (UniqueName: \"kubernetes.io/projected/bd39ad63-ddfa-42ef-8c17-c35cef29961e-kube-api-access-bs47v\") pod \"maas-api-revoke-sub-1e7ef370-wbgt7\" (UID: \"bd39ad63-ddfa-42ef-8c17-c35cef29961e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7ef370-wbgt7" Nov 03 00:46:59.085287 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:59.085249 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7ef370-wbgt7" Nov 03 00:46:59.457197 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:59.457167 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e7ef370-wbgt7"] Nov 03 00:46:59.459439 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:46:59.459411 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd39ad63_ddfa_42ef_8c17_c35cef29961e.slice/crio-adfcca2f27976c418108352a4ab0646ca4da7d20cb926c8d28f19343549f1745 WatchSource:0}: Error finding container adfcca2f27976c418108352a4ab0646ca4da7d20cb926c8d28f19343549f1745: Status 404 returned error can't find the container with id adfcca2f27976c418108352a4ab0646ca4da7d20cb926c8d28f19343549f1745 Nov 03 00:46:59.618443 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:46:59.618402 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7ef370-wbgt7" event={"ID":"bd39ad63-ddfa-42ef-8c17-c35cef29961e","Type":"ContainerStarted","Data":"adfcca2f27976c418108352a4ab0646ca4da7d20cb926c8d28f19343549f1745"} Nov 03 00:47:00.624837 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:00.624801 5114 generic.go:356] "Generic (PLEG): container finished" podID="bd39ad63-ddfa-42ef-8c17-c35cef29961e" containerID="544d413450a13ebd8b20a631bf2f6fd79bf1e9b5edb738857b9bac1915fc9f6d" exitCode=0 Nov 03 00:47:00.625288 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:00.624891 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7ef370-wbgt7" event={"ID":"bd39ad63-ddfa-42ef-8c17-c35cef29961e","Type":"ContainerDied","Data":"544d413450a13ebd8b20a631bf2f6fd79bf1e9b5edb738857b9bac1915fc9f6d"} Nov 03 00:47:01.774283 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:01.774244 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7ef370-wbgt7" Nov 03 00:47:01.880370 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:01.880273 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd39ad63-ddfa-42ef-8c17-c35cef29961e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd39ad63-ddfa-42ef-8c17-c35cef29961e-maas-api-xks-ca\") pod \"bd39ad63-ddfa-42ef-8c17-c35cef29961e\" (UID: \"bd39ad63-ddfa-42ef-8c17-c35cef29961e\") " Nov 03 00:47:01.880370 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:01.880329 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd39ad63-ddfa-42ef-8c17-c35cef29961e-kube-api-access-bs47v\" (UniqueName: \"kubernetes.io/projected/bd39ad63-ddfa-42ef-8c17-c35cef29961e-kube-api-access-bs47v\") pod \"bd39ad63-ddfa-42ef-8c17-c35cef29961e\" (UID: \"bd39ad63-ddfa-42ef-8c17-c35cef29961e\") " Nov 03 00:47:01.880370 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:01.880364 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd39ad63-ddfa-42ef-8c17-c35cef29961e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd39ad63-ddfa-42ef-8c17-c35cef29961e-maas-api-openshift-ca\") pod \"bd39ad63-ddfa-42ef-8c17-c35cef29961e\" (UID: \"bd39ad63-ddfa-42ef-8c17-c35cef29961e\") " Nov 03 00:47:01.881071 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:01.880884 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd39ad63-ddfa-42ef-8c17-c35cef29961e-maas-api-openshift-ca" pod "bd39ad63-ddfa-42ef-8c17-c35cef29961e" (UID: "bd39ad63-ddfa-42ef-8c17-c35cef29961e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:47:01.881217 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:01.881191 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd39ad63-ddfa-42ef-8c17-c35cef29961e-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:01.882981 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:01.882949 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd39ad63-ddfa-42ef-8c17-c35cef29961e-kube-api-access-bs47v" pod "bd39ad63-ddfa-42ef-8c17-c35cef29961e" (UID: "bd39ad63-ddfa-42ef-8c17-c35cef29961e"). InnerVolumeSpecName "kube-api-access-bs47v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:47:01.883168 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:01.883146 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd39ad63-ddfa-42ef-8c17-c35cef29961e-maas-api-xks-ca" pod "bd39ad63-ddfa-42ef-8c17-c35cef29961e" (UID: "bd39ad63-ddfa-42ef-8c17-c35cef29961e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:47:01.982122 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:01.982067 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd39ad63-ddfa-42ef-8c17-c35cef29961e-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:01.982122 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:01.982121 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs47v\" (UniqueName: \"kubernetes.io/projected/bd39ad63-ddfa-42ef-8c17-c35cef29961e-kube-api-access-bs47v\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:02.654525 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:02.654150 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7ef370-wbgt7" event={"ID":"bd39ad63-ddfa-42ef-8c17-c35cef29961e","Type":"ContainerDied","Data":"adfcca2f27976c418108352a4ab0646ca4da7d20cb926c8d28f19343549f1745"} Nov 03 00:47:02.654525 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:02.654223 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adfcca2f27976c418108352a4ab0646ca4da7d20cb926c8d28f19343549f1745" Nov 03 00:47:02.654525 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:02.654301 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e7ef370-wbgt7" Nov 03 00:47:06.046105 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:06.046065 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd3759ee-rs6zs"] Nov 03 00:47:06.095829 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:06.095790 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd3759ee-rs6zs"] Nov 03 00:47:06.792030 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:06.791989 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4da03fdb-612b-4dc9-9244-439b9073a10f" path="/var/lib/kubelet/pods/4da03fdb-612b-4dc9-9244-439b9073a10f/volumes" Nov 03 00:47:07.584893 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:47:07.584851 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:47:09.521178 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:09.521134 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1435c62-hskbc"] Nov 03 00:47:09.521635 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:09.521505 5114 cpu_manager.go:414] "containerMap: removing container" podUID="bd39ad63-ddfa-42ef-8c17-c35cef29961e" containerName="revoke-keys" Nov 03 00:47:09.521635 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:09.521517 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd39ad63-ddfa-42ef-8c17-c35cef29961e" containerName="revoke-keys" Nov 03 00:47:09.521635 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:09.521569 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd39ad63-ddfa-42ef-8c17-c35cef29961e" containerName="revoke-keys" Nov 03 00:47:09.524856 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:09.524838 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1435c62-hskbc" Nov 03 00:47:09.553433 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:09.553398 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/924a15c6-5f81-4a1b-8bc2-553467c528c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1435c62-hskbc\" (UID: \"924a15c6-5f81-4a1b-8bc2-553467c528c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1435c62-hskbc" Nov 03 00:47:09.553681 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:09.553465 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s4n2\" (UniqueName: \"kubernetes.io/projected/924a15c6-5f81-4a1b-8bc2-553467c528c8-kube-api-access-7s4n2\") pod \"maas-api-revoke-sub-a1435c62-hskbc\" (UID: \"924a15c6-5f81-4a1b-8bc2-553467c528c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1435c62-hskbc" Nov 03 00:47:09.553681 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:09.553506 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/924a15c6-5f81-4a1b-8bc2-553467c528c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1435c62-hskbc\" (UID: \"924a15c6-5f81-4a1b-8bc2-553467c528c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1435c62-hskbc" Nov 03 00:47:09.577002 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:09.576956 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:47:09.577002 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:09.576967 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:47:09.626516 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:09.626450 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:47:09.626718 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:09.626526 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:47:09.628368 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:09.628348 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1435c62-hskbc"] Nov 03 00:47:09.654427 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:09.654382 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/924a15c6-5f81-4a1b-8bc2-553467c528c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1435c62-hskbc\" (UID: \"924a15c6-5f81-4a1b-8bc2-553467c528c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1435c62-hskbc" Nov 03 00:47:09.654636 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:09.654469 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s4n2\" (UniqueName: \"kubernetes.io/projected/924a15c6-5f81-4a1b-8bc2-553467c528c8-kube-api-access-7s4n2\") pod \"maas-api-revoke-sub-a1435c62-hskbc\" (UID: \"924a15c6-5f81-4a1b-8bc2-553467c528c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1435c62-hskbc" Nov 03 00:47:09.654636 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:09.654541 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/924a15c6-5f81-4a1b-8bc2-553467c528c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1435c62-hskbc\" (UID: \"924a15c6-5f81-4a1b-8bc2-553467c528c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1435c62-hskbc" Nov 03 00:47:09.655269 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:09.655249 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/924a15c6-5f81-4a1b-8bc2-553467c528c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1435c62-hskbc\" (UID: \"924a15c6-5f81-4a1b-8bc2-553467c528c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1435c62-hskbc" Nov 03 00:47:09.657002 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:09.656960 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/924a15c6-5f81-4a1b-8bc2-553467c528c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1435c62-hskbc\" (UID: \"924a15c6-5f81-4a1b-8bc2-553467c528c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1435c62-hskbc" Nov 03 00:47:09.715166 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:09.715126 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s4n2\" (UniqueName: \"kubernetes.io/projected/924a15c6-5f81-4a1b-8bc2-553467c528c8-kube-api-access-7s4n2\") pod \"maas-api-revoke-sub-a1435c62-hskbc\" (UID: \"924a15c6-5f81-4a1b-8bc2-553467c528c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1435c62-hskbc" Nov 03 00:47:09.837553 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:09.837515 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1435c62-hskbc" Nov 03 00:47:10.210782 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:10.210757 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1435c62-hskbc"] Nov 03 00:47:10.212744 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:47:10.212714 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924a15c6_5f81_4a1b_8bc2_553467c528c8.slice/crio-fde72eac625459a64e4be92b91a0363c0118f6eee2255ecbb2b4124df6d0b49d WatchSource:0}: Error finding container fde72eac625459a64e4be92b91a0363c0118f6eee2255ecbb2b4124df6d0b49d: Status 404 returned error can't find the container with id fde72eac625459a64e4be92b91a0363c0118f6eee2255ecbb2b4124df6d0b49d Nov 03 00:47:10.711341 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:10.711302 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1435c62-hskbc" event={"ID":"924a15c6-5f81-4a1b-8bc2-553467c528c8","Type":"ContainerStarted","Data":"ac07318e32790ad2f2e20311858ecd14ff93ef5735061fbeb013d6e4496747ea"} Nov 03 00:47:10.711790 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:10.711393 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1435c62-hskbc" event={"ID":"924a15c6-5f81-4a1b-8bc2-553467c528c8","Type":"ContainerStarted","Data":"fde72eac625459a64e4be92b91a0363c0118f6eee2255ecbb2b4124df6d0b49d"} Nov 03 00:47:10.827413 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:10.827346 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1435c62-hskbc" podStartSLOduration=1.8273310170000001 podStartE2EDuration="1.827331017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:10.825871086 +0000 UTC m=+7106.178904458" watchObservedRunningTime="2026-11-03 00:47:10.827331017 +0000 UTC m=+7106.180364389" Nov 03 00:47:11.717820 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:11.717785 5114 generic.go:356] "Generic (PLEG): container finished" podID="924a15c6-5f81-4a1b-8bc2-553467c528c8" containerID="ac07318e32790ad2f2e20311858ecd14ff93ef5735061fbeb013d6e4496747ea" exitCode=0 Nov 03 00:47:11.718252 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:11.717837 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1435c62-hskbc" event={"ID":"924a15c6-5f81-4a1b-8bc2-553467c528c8","Type":"ContainerDied","Data":"ac07318e32790ad2f2e20311858ecd14ff93ef5735061fbeb013d6e4496747ea"} Nov 03 00:47:12.885082 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:12.885045 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1435c62-hskbc" Nov 03 00:47:12.984982 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:12.984864 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/924a15c6-5f81-4a1b-8bc2-553467c528c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/924a15c6-5f81-4a1b-8bc2-553467c528c8-maas-api-xks-ca\") pod \"924a15c6-5f81-4a1b-8bc2-553467c528c8\" (UID: \"924a15c6-5f81-4a1b-8bc2-553467c528c8\") " Nov 03 00:47:12.985196 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:12.985007 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924a15c6-5f81-4a1b-8bc2-553467c528c8-kube-api-access-7s4n2\" (UniqueName: \"kubernetes.io/projected/924a15c6-5f81-4a1b-8bc2-553467c528c8-kube-api-access-7s4n2\") pod \"924a15c6-5f81-4a1b-8bc2-553467c528c8\" (UID: \"924a15c6-5f81-4a1b-8bc2-553467c528c8\") " Nov 03 00:47:12.985196 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:12.985052 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/924a15c6-5f81-4a1b-8bc2-553467c528c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/924a15c6-5f81-4a1b-8bc2-553467c528c8-maas-api-openshift-ca\") pod \"924a15c6-5f81-4a1b-8bc2-553467c528c8\" (UID: \"924a15c6-5f81-4a1b-8bc2-553467c528c8\") " Nov 03 00:47:12.985793 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:12.985757 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924a15c6-5f81-4a1b-8bc2-553467c528c8-maas-api-openshift-ca" pod "924a15c6-5f81-4a1b-8bc2-553467c528c8" (UID: "924a15c6-5f81-4a1b-8bc2-553467c528c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:47:12.988599 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:12.988553 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/924a15c6-5f81-4a1b-8bc2-553467c528c8-maas-api-xks-ca" pod "924a15c6-5f81-4a1b-8bc2-553467c528c8" (UID: "924a15c6-5f81-4a1b-8bc2-553467c528c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:47:12.988863 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:12.988829 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924a15c6-5f81-4a1b-8bc2-553467c528c8-kube-api-access-7s4n2" pod "924a15c6-5f81-4a1b-8bc2-553467c528c8" (UID: "924a15c6-5f81-4a1b-8bc2-553467c528c8"). InnerVolumeSpecName "kube-api-access-7s4n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:47:13.088186 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:13.088074 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/924a15c6-5f81-4a1b-8bc2-553467c528c8-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:13.088186 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:13.088126 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s4n2\" (UniqueName: \"kubernetes.io/projected/924a15c6-5f81-4a1b-8bc2-553467c528c8-kube-api-access-7s4n2\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:13.088186 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:13.088145 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/924a15c6-5f81-4a1b-8bc2-553467c528c8-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:13.750511 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:13.750225 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1435c62-hskbc" event={"ID":"924a15c6-5f81-4a1b-8bc2-553467c528c8","Type":"ContainerDied","Data":"fde72eac625459a64e4be92b91a0363c0118f6eee2255ecbb2b4124df6d0b49d"} Nov 03 00:47:13.750511 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:13.750293 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fde72eac625459a64e4be92b91a0363c0118f6eee2255ecbb2b4124df6d0b49d" Nov 03 00:47:13.750511 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:13.750365 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1435c62-hskbc" Nov 03 00:47:20.318262 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:20.318224 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c204b05f-4p6pb"] Nov 03 00:47:20.318814 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:20.318601 5114 cpu_manager.go:414] "containerMap: removing container" podUID="924a15c6-5f81-4a1b-8bc2-553467c528c8" containerName="revoke-keys" Nov 03 00:47:20.318814 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:20.318614 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="924a15c6-5f81-4a1b-8bc2-553467c528c8" containerName="revoke-keys" Nov 03 00:47:20.318814 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:20.318663 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="924a15c6-5f81-4a1b-8bc2-553467c528c8" containerName="revoke-keys" Nov 03 00:47:20.321373 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:20.321355 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c204b05f-4p6pb" Nov 03 00:47:20.352708 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:20.352668 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/751d3774-0178-4c81-a5f1-b56f875c23ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c204b05f-4p6pb\" (UID: \"751d3774-0178-4c81-a5f1-b56f875c23ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c204b05f-4p6pb" Nov 03 00:47:20.352708 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:20.352712 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l967g\" (UniqueName: \"kubernetes.io/projected/751d3774-0178-4c81-a5f1-b56f875c23ce-kube-api-access-l967g\") pod \"maas-api-revoke-sub-c204b05f-4p6pb\" (UID: \"751d3774-0178-4c81-a5f1-b56f875c23ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c204b05f-4p6pb" Nov 03 00:47:20.352994 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:20.352742 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/751d3774-0178-4c81-a5f1-b56f875c23ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c204b05f-4p6pb\" (UID: \"751d3774-0178-4c81-a5f1-b56f875c23ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c204b05f-4p6pb" Nov 03 00:47:20.373733 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:20.373695 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:47:20.423127 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:20.423092 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:47:20.423364 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:20.423142 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:47:20.423364 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:20.423201 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:47:20.425296 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:20.425270 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c204b05f-4p6pb"] Nov 03 00:47:20.453502 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:20.453451 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/751d3774-0178-4c81-a5f1-b56f875c23ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c204b05f-4p6pb\" (UID: \"751d3774-0178-4c81-a5f1-b56f875c23ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c204b05f-4p6pb" Nov 03 00:47:20.453706 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:20.453527 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l967g\" (UniqueName: \"kubernetes.io/projected/751d3774-0178-4c81-a5f1-b56f875c23ce-kube-api-access-l967g\") pod \"maas-api-revoke-sub-c204b05f-4p6pb\" (UID: \"751d3774-0178-4c81-a5f1-b56f875c23ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c204b05f-4p6pb" Nov 03 00:47:20.453706 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:20.453553 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/751d3774-0178-4c81-a5f1-b56f875c23ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c204b05f-4p6pb\" (UID: \"751d3774-0178-4c81-a5f1-b56f875c23ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c204b05f-4p6pb" Nov 03 00:47:20.454184 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:20.454163 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/751d3774-0178-4c81-a5f1-b56f875c23ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c204b05f-4p6pb\" (UID: \"751d3774-0178-4c81-a5f1-b56f875c23ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c204b05f-4p6pb" Nov 03 00:47:20.456116 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:20.456089 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/751d3774-0178-4c81-a5f1-b56f875c23ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c204b05f-4p6pb\" (UID: \"751d3774-0178-4c81-a5f1-b56f875c23ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c204b05f-4p6pb" Nov 03 00:47:20.514682 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:20.514640 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l967g\" (UniqueName: \"kubernetes.io/projected/751d3774-0178-4c81-a5f1-b56f875c23ce-kube-api-access-l967g\") pod \"maas-api-revoke-sub-c204b05f-4p6pb\" (UID: \"751d3774-0178-4c81-a5f1-b56f875c23ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c204b05f-4p6pb" Nov 03 00:47:20.634413 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:20.634319 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c204b05f-4p6pb" Nov 03 00:47:21.008705 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:21.008679 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c204b05f-4p6pb"] Nov 03 00:47:21.010334 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:47:21.010298 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod751d3774_0178_4c81_a5f1_b56f875c23ce.slice/crio-037d84c5f15088d08d0b6c27ec45d825827fcad1f75b0c6dd2cc83f05afdf268 WatchSource:0}: Error finding container 037d84c5f15088d08d0b6c27ec45d825827fcad1f75b0c6dd2cc83f05afdf268: Status 404 returned error can't find the container with id 037d84c5f15088d08d0b6c27ec45d825827fcad1f75b0c6dd2cc83f05afdf268 Nov 03 00:47:21.794847 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:21.794814 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c204b05f-4p6pb" event={"ID":"751d3774-0178-4c81-a5f1-b56f875c23ce","Type":"ContainerStarted","Data":"58d19d4f679b63502a83ce19d89556b5bb06e8f6a2e7faded663c1768885306e"} Nov 03 00:47:21.795400 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:21.794880 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c204b05f-4p6pb" event={"ID":"751d3774-0178-4c81-a5f1-b56f875c23ce","Type":"ContainerStarted","Data":"037d84c5f15088d08d0b6c27ec45d825827fcad1f75b0c6dd2cc83f05afdf268"} Nov 03 00:47:21.911666 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:21.911596 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c204b05f-4p6pb" podStartSLOduration=1.9115784059999998 podStartE2EDuration="1.911578406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:21.90985738 +0000 UTC m=+7117.262890755" watchObservedRunningTime="2026-11-03 00:47:21.911578406 +0000 UTC m=+7117.264611844" Nov 03 00:47:22.801030 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:22.800988 5114 generic.go:356] "Generic (PLEG): container finished" podID="751d3774-0178-4c81-a5f1-b56f875c23ce" containerID="58d19d4f679b63502a83ce19d89556b5bb06e8f6a2e7faded663c1768885306e" exitCode=0 Nov 03 00:47:22.801466 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:22.801080 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c204b05f-4p6pb" event={"ID":"751d3774-0178-4c81-a5f1-b56f875c23ce","Type":"ContainerDied","Data":"58d19d4f679b63502a83ce19d89556b5bb06e8f6a2e7faded663c1768885306e"} Nov 03 00:47:24.168970 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:24.168938 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c204b05f-4p6pb" Nov 03 00:47:24.195889 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:24.195849 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/751d3774-0178-4c81-a5f1-b56f875c23ce-kube-api-access-l967g\" (UniqueName: \"kubernetes.io/projected/751d3774-0178-4c81-a5f1-b56f875c23ce-kube-api-access-l967g\") pod \"751d3774-0178-4c81-a5f1-b56f875c23ce\" (UID: \"751d3774-0178-4c81-a5f1-b56f875c23ce\") " Nov 03 00:47:24.196083 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:24.195909 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/751d3774-0178-4c81-a5f1-b56f875c23ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/751d3774-0178-4c81-a5f1-b56f875c23ce-maas-api-xks-ca\") pod \"751d3774-0178-4c81-a5f1-b56f875c23ce\" (UID: \"751d3774-0178-4c81-a5f1-b56f875c23ce\") " Nov 03 00:47:24.196083 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:24.195939 5114 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/751d3774-0178-4c81-a5f1-b56f875c23ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/751d3774-0178-4c81-a5f1-b56f875c23ce-maas-api-openshift-ca\") pod \"751d3774-0178-4c81-a5f1-b56f875c23ce\" (UID: \"751d3774-0178-4c81-a5f1-b56f875c23ce\") " Nov 03 00:47:24.196327 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:24.196293 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/751d3774-0178-4c81-a5f1-b56f875c23ce-maas-api-openshift-ca" pod "751d3774-0178-4c81-a5f1-b56f875c23ce" (UID: "751d3774-0178-4c81-a5f1-b56f875c23ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:47:24.198282 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:24.198253 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/751d3774-0178-4c81-a5f1-b56f875c23ce-maas-api-xks-ca" pod "751d3774-0178-4c81-a5f1-b56f875c23ce" (UID: "751d3774-0178-4c81-a5f1-b56f875c23ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:47:24.198282 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:24.198279 5114 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/751d3774-0178-4c81-a5f1-b56f875c23ce-kube-api-access-l967g" pod "751d3774-0178-4c81-a5f1-b56f875c23ce" (UID: "751d3774-0178-4c81-a5f1-b56f875c23ce"). InnerVolumeSpecName "kube-api-access-l967g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:47:24.297324 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:24.297241 5114 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l967g\" (UniqueName: \"kubernetes.io/projected/751d3774-0178-4c81-a5f1-b56f875c23ce-kube-api-access-l967g\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:24.297324 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:24.297272 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/751d3774-0178-4c81-a5f1-b56f875c23ce-maas-api-xks-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:24.297324 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:24.297284 5114 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/751d3774-0178-4c81-a5f1-b56f875c23ce-maas-api-openshift-ca\") on node \"ip-10-0-135-250.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:24.813369 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:24.813333 5114 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c204b05f-4p6pb" Nov 03 00:47:24.813369 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:24.813355 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c204b05f-4p6pb" event={"ID":"751d3774-0178-4c81-a5f1-b56f875c23ce","Type":"ContainerDied","Data":"037d84c5f15088d08d0b6c27ec45d825827fcad1f75b0c6dd2cc83f05afdf268"} Nov 03 00:47:24.813634 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:24.813409 5114 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="037d84c5f15088d08d0b6c27ec45d825827fcad1f75b0c6dd2cc83f05afdf268" Nov 03 00:47:37.583389 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:47:37.583347 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:47:42.047868 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:42.047816 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f1b7d9f-rbxjm"] Nov 03 00:47:42.097534 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:42.097476 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f1b7d9f-rbxjm"] Nov 03 00:47:42.791628 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:47:42.791586 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="603ad53d-b227-4870-95da-99ef1a795914" path="/var/lib/kubelet/pods/603ad53d-b227-4870-95da-99ef1a795914/volumes" Nov 03 00:48:00.356851 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:00.356817 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-d6fmn_289a684e-75c9-4b50-ad59-0ab341e55c61/manager/0.log" Nov 03 00:48:01.395518 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:01.395463 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-bwb4h_48efe895-cacd-4880-94d4-3d965679e301/manager/0.log" Nov 03 00:48:03.292492 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:03.292448 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-16c84053-tp5l9_ed01776b-287a-4d6a-8897-9b65df4b9cfe/revoke-keys/0.log" Nov 03 00:48:03.510698 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:03.510667 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1c4e171f-nlvvf_64a1343f-c07d-4c54-9fdf-28f2e20c4caf/revoke-keys/0.log" Nov 03 00:48:03.728845 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:03.728766 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1e7ef370-wbgt7_bd39ad63-ddfa-42ef-8c17-c35cef29961e/revoke-keys/0.log" Nov 03 00:48:03.938991 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:03.938962 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2ce57900-d2x7m_7445cbe5-60fd-4c6e-a2a7-19161eac085a/revoke-keys/0.log" Nov 03 00:48:04.181798 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:04.181765 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2ec1f0a2-b7qmq_b7b5831e-4ed3-42db-8c3e-1236ccd8d122/revoke-keys/0.log" Nov 03 00:48:04.399567 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:04.399536 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-443a2b88-qfkfq_e7961012-23c1-458a-83fe-0354645dc03a/revoke-keys/0.log" Nov 03 00:48:04.618407 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:04.618375 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4e265255-f5t8n_743bbffc-bba5-4154-8f25-3aa2ce6784c3/revoke-keys/0.log" Nov 03 00:48:04.833681 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:04.833650 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6d0d532c-6bxv7_07397238-7175-458f-9965-087261e39f88/revoke-keys/0.log" Nov 03 00:48:05.044609 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:05.044577 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8b6f0b52-5vbg5_d01e42a2-9e61-48b4-b304-030a63f67f0d/revoke-keys/0.log" Nov 03 00:48:05.265129 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:05.265086 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a1435c62-hskbc_924a15c6-5f81-4a1b-8bc2-553467c528c8/revoke-keys/0.log" Nov 03 00:48:05.488876 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:05.488796 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c204b05f-4p6pb_751d3774-0178-4c81-a5f1-b56f875c23ce/revoke-keys/0.log" Nov 03 00:48:05.704230 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:05.704202 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c7de5d31-zhthq_0dd10eac-9837-4539-b484-f858468ef5cf/revoke-keys/0.log" Nov 03 00:48:05.924278 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:05.924250 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c8e24a2c-6zhhh_44b48023-8186-4e7a-9cb8-2b4950e796b7/revoke-keys/0.log" Nov 03 00:48:06.137675 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:06.137644 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e5a01859-j64l6_983a5673-f7b4-4c32-830a-daa08ffd4ea4/revoke-keys/0.log" Nov 03 00:48:06.355454 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:06.355428 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e7a31d8e-8dgpv_bb802929-b2d0-4191-9f35-4cfee6005793/revoke-keys/0.log" Nov 03 00:48:06.572248 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:06.572218 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ee7cc6e5-2696t_ba20182f-8631-473a-b67f-91067e66f621/revoke-keys/0.log" Nov 03 00:48:07.583005 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:48:07.582965 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:48:12.248108 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:12.248077 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65dbc8865f-58bxj_247dce35-9881-472f-b000-686c2a0a5dfa/kube-auth-proxy/0.log" Nov 03 00:48:12.979206 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:12.979177 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-7zbtz_19ba6ae1-0768-4641-9026-bce647422531/istio-proxy/0.log" Nov 03 00:48:13.575266 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:13.575234 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-92jjh_b4e963bb-8c2f-4628-a009-ebbebd62394f/istio-proxy/0.log" Nov 03 00:48:14.453313 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:14.453276 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-ksnhk_713030b6-6b1c-4cf9-b01a-58171f6ad0c6/istio-proxy/0.log" Nov 03 00:48:18.971315 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:18.971281 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-6ttq4_832b80ef-5c2c-4791-81c4-b92748d79552/main/0.log" Nov 03 00:48:26.057819 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:26.057781 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-443a2b88-qfkfq"] Nov 03 00:48:26.107292 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:26.107253 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-443a2b88-qfkfq"] Nov 03 00:48:26.792383 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:26.792338 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7961012-23c1-458a-83fe-0354645dc03a" path="/var/lib/kubelet/pods/e7961012-23c1-458a-83fe-0354645dc03a/volumes" Nov 03 00:48:29.050601 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.050559 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xzj92/must-gather-nmsjr"] Nov 03 00:48:29.051118 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.050984 5114 cpu_manager.go:414] "containerMap: removing container" podUID="751d3774-0178-4c81-a5f1-b56f875c23ce" containerName="revoke-keys" Nov 03 00:48:29.051118 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.050998 5114 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="751d3774-0178-4c81-a5f1-b56f875c23ce" containerName="revoke-keys" Nov 03 00:48:29.051118 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.051074 5114 memory_manager.go:365] "RemoveStaleState removing state" podUID="751d3774-0178-4c81-a5f1-b56f875c23ce" containerName="revoke-keys" Nov 03 00:48:29.054558 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.054536 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xzj92/must-gather-nmsjr" Nov 03 00:48:29.091979 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.091941 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm9t9\" (UniqueName: \"kubernetes.io/projected/5b5cb1d6-0b97-4468-a494-0f241acd9cfa-kube-api-access-wm9t9\") pod \"must-gather-nmsjr\" (UID: \"5b5cb1d6-0b97-4468-a494-0f241acd9cfa\") " pod="openshift-must-gather-xzj92/must-gather-nmsjr" Nov 03 00:48:29.092206 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.091999 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5b5cb1d6-0b97-4468-a494-0f241acd9cfa-must-gather-output\") pod \"must-gather-nmsjr\" (UID: \"5b5cb1d6-0b97-4468-a494-0f241acd9cfa\") " pod="openshift-must-gather-xzj92/must-gather-nmsjr" Nov 03 00:48:29.106780 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.106745 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xzj92\"/\"kube-root-ca.crt\"" Nov 03 00:48:29.106997 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.106832 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xzj92\"/\"openshift-service-ca.crt\"" Nov 03 00:48:29.156021 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.155983 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xzj92\"/\"default-dockercfg-rkc5z\"" Nov 03 00:48:29.159587 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.159553 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xzj92/must-gather-nmsjr"] Nov 03 00:48:29.162587 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.162560 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t6hpv/must-gather-swjbq"] Nov 03 00:48:29.166788 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.166767 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t6hpv/must-gather-swjbq" Nov 03 00:48:29.193163 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.193128 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm9t9\" (UniqueName: \"kubernetes.io/projected/5b5cb1d6-0b97-4468-a494-0f241acd9cfa-kube-api-access-wm9t9\") pod \"must-gather-nmsjr\" (UID: \"5b5cb1d6-0b97-4468-a494-0f241acd9cfa\") " pod="openshift-must-gather-xzj92/must-gather-nmsjr" Nov 03 00:48:29.193401 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.193182 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxzbq\" (UniqueName: \"kubernetes.io/projected/a2be6f06-1dd6-4c66-87f1-2432ba9ed5d6-kube-api-access-lxzbq\") pod \"must-gather-swjbq\" (UID: \"a2be6f06-1dd6-4c66-87f1-2432ba9ed5d6\") " pod="openshift-must-gather-t6hpv/must-gather-swjbq" Nov 03 00:48:29.193401 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.193213 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a2be6f06-1dd6-4c66-87f1-2432ba9ed5d6-must-gather-output\") pod \"must-gather-swjbq\" (UID: \"a2be6f06-1dd6-4c66-87f1-2432ba9ed5d6\") " pod="openshift-must-gather-t6hpv/must-gather-swjbq" Nov 03 00:48:29.193401 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.193259 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5b5cb1d6-0b97-4468-a494-0f241acd9cfa-must-gather-output\") pod \"must-gather-nmsjr\" (UID: \"5b5cb1d6-0b97-4468-a494-0f241acd9cfa\") " pod="openshift-must-gather-xzj92/must-gather-nmsjr" Nov 03 00:48:29.193723 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.193699 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5b5cb1d6-0b97-4468-a494-0f241acd9cfa-must-gather-output\") pod \"must-gather-nmsjr\" (UID: \"5b5cb1d6-0b97-4468-a494-0f241acd9cfa\") " pod="openshift-must-gather-xzj92/must-gather-nmsjr" Nov 03 00:48:29.210887 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.210848 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t6hpv/must-gather-swjbq"] Nov 03 00:48:29.218952 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.218924 5114 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t6hpv\"/\"default-dockercfg-pwrpp\"" Nov 03 00:48:29.219138 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.219040 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t6hpv\"/\"openshift-service-ca.crt\"" Nov 03 00:48:29.219138 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.219095 5114 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t6hpv\"/\"kube-root-ca.crt\"" Nov 03 00:48:29.254231 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.254198 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm9t9\" (UniqueName: \"kubernetes.io/projected/5b5cb1d6-0b97-4468-a494-0f241acd9cfa-kube-api-access-wm9t9\") pod \"must-gather-nmsjr\" (UID: \"5b5cb1d6-0b97-4468-a494-0f241acd9cfa\") " pod="openshift-must-gather-xzj92/must-gather-nmsjr" Nov 03 00:48:29.294508 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.294437 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxzbq\" (UniqueName: \"kubernetes.io/projected/a2be6f06-1dd6-4c66-87f1-2432ba9ed5d6-kube-api-access-lxzbq\") pod \"must-gather-swjbq\" (UID: \"a2be6f06-1dd6-4c66-87f1-2432ba9ed5d6\") " pod="openshift-must-gather-t6hpv/must-gather-swjbq" Nov 03 00:48:29.294508 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.294474 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a2be6f06-1dd6-4c66-87f1-2432ba9ed5d6-must-gather-output\") pod \"must-gather-swjbq\" (UID: \"a2be6f06-1dd6-4c66-87f1-2432ba9ed5d6\") " pod="openshift-must-gather-t6hpv/must-gather-swjbq" Nov 03 00:48:29.294901 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.294881 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a2be6f06-1dd6-4c66-87f1-2432ba9ed5d6-must-gather-output\") pod \"must-gather-swjbq\" (UID: \"a2be6f06-1dd6-4c66-87f1-2432ba9ed5d6\") " pod="openshift-must-gather-t6hpv/must-gather-swjbq" Nov 03 00:48:29.364377 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.364301 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxzbq\" (UniqueName: \"kubernetes.io/projected/a2be6f06-1dd6-4c66-87f1-2432ba9ed5d6-kube-api-access-lxzbq\") pod \"must-gather-swjbq\" (UID: \"a2be6f06-1dd6-4c66-87f1-2432ba9ed5d6\") " pod="openshift-must-gather-t6hpv/must-gather-swjbq" Nov 03 00:48:29.368197 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.368168 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xzj92/must-gather-nmsjr" Nov 03 00:48:29.484002 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.483959 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t6hpv/must-gather-swjbq" Nov 03 00:48:29.745438 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.745326 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xzj92/must-gather-nmsjr"] Nov 03 00:48:29.748208 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:48:29.748178 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b5cb1d6_0b97_4468_a494_0f241acd9cfa.slice/crio-3419bbdd71ddc23b32be65f8d3fd91b8141456b4456c6ad630a4ddc935b58d0a WatchSource:0}: Error finding container 3419bbdd71ddc23b32be65f8d3fd91b8141456b4456c6ad630a4ddc935b58d0a: Status 404 returned error can't find the container with id 3419bbdd71ddc23b32be65f8d3fd91b8141456b4456c6ad630a4ddc935b58d0a Nov 03 00:48:29.750058 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.750038 5114 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:48:29.861711 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:29.861681 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t6hpv/must-gather-swjbq"] Nov 03 00:48:29.863848 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:48:29.863817 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2be6f06_1dd6_4c66_87f1_2432ba9ed5d6.slice/crio-85132a24432da56b2d439adb8dbc6c1fb965ad142a281eee5d362bae474f0c1b WatchSource:0}: Error finding container 85132a24432da56b2d439adb8dbc6c1fb965ad142a281eee5d362bae474f0c1b: Status 404 returned error can't find the container with id 85132a24432da56b2d439adb8dbc6c1fb965ad142a281eee5d362bae474f0c1b Nov 03 00:48:30.163695 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:30.163658 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t6hpv/must-gather-swjbq" event={"ID":"a2be6f06-1dd6-4c66-87f1-2432ba9ed5d6","Type":"ContainerStarted","Data":"85132a24432da56b2d439adb8dbc6c1fb965ad142a281eee5d362bae474f0c1b"} Nov 03 00:48:30.164913 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:30.164888 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xzj92/must-gather-nmsjr" event={"ID":"5b5cb1d6-0b97-4468-a494-0f241acd9cfa","Type":"ContainerStarted","Data":"3419bbdd71ddc23b32be65f8d3fd91b8141456b4456c6ad630a4ddc935b58d0a"} Nov 03 00:48:32.191119 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:32.191069 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t6hpv/must-gather-swjbq" event={"ID":"a2be6f06-1dd6-4c66-87f1-2432ba9ed5d6","Type":"ContainerStarted","Data":"16a229c135b654f69d6c84dc79adb10c91449ddce1da23f99d0b7f0dcd157912"} Nov 03 00:48:32.191780 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:32.191138 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t6hpv/must-gather-swjbq" event={"ID":"a2be6f06-1dd6-4c66-87f1-2432ba9ed5d6","Type":"ContainerStarted","Data":"e16aa7b2b4a407ab9f618dfd7b812d728d43d7929648f87a5133a9163eb27ab3"} Nov 03 00:48:32.307126 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:32.307056 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-t6hpv/must-gather-swjbq" podStartSLOduration=2.000452103 podStartE2EDuration="3.307032765s" totalImagesPullingTime="1.306580662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:32.305467907 +0000 UTC m=+7187.658501296" watchObservedRunningTime="2026-11-03 00:48:32.307032765 +0000 UTC m=+7187.660066141" Nov 03 00:48:37.047393 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:37.047347 5114 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7de5d31-zhthq"] Nov 03 00:48:37.096871 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:37.096829 5114 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7de5d31-zhthq"] Nov 03 00:48:37.590016 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:48:37.589972 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:48:38.231685 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:38.231590 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xzj92/must-gather-nmsjr" event={"ID":"5b5cb1d6-0b97-4468-a494-0f241acd9cfa","Type":"ContainerStarted","Data":"7c7c496b960b2b352a9e78e39bee679f5d45a0c3a3c2c78d45fa33273e739273"} Nov 03 00:48:38.231685 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:38.231641 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xzj92/must-gather-nmsjr" event={"ID":"5b5cb1d6-0b97-4468-a494-0f241acd9cfa","Type":"ContainerStarted","Data":"73a64badd23b12516435384e77524830fb6e0c6e6b3f02c8ddaf7e010aacb0d1"} Nov 03 00:48:38.348983 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:38.348909 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-xzj92/must-gather-nmsjr" podStartSLOduration=1.288411934 podStartE2EDuration="9.348889832s" totalImagesPullingTime="8.060477898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:38.346331238 +0000 UTC m=+7193.699364615" watchObservedRunningTime="2026-11-03 00:48:38.348889832 +0000 UTC m=+7193.701923202" Nov 03 00:48:38.396245 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:38.396202 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9drjv_e608a3bc-a7f2-45a1-9603-d85bbbe20614/global-pull-secret-syncer/0.log" Nov 03 00:48:38.806506 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:38.802558 5114 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd10eac-9837-4539-b484-f858468ef5cf" path="/var/lib/kubelet/pods/0dd10eac-9837-4539-b484-f858468ef5cf/volumes" Nov 03 00:48:42.190505 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:42.189977 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ftp5v_1062888a-e10c-4a34-b84a-f85ff9eb8063/konnectivity-agent/0.log" Nov 03 00:48:43.811477 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:43.811447 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-250.us-west-1.compute.internal_09bea7df1640e9a4e82191575067a028/haproxy/0.log" Nov 03 00:48:44.660922 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:44.660772 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-d6fmn_289a684e-75c9-4b50-ad59-0ab341e55c61/manager/0.log" Nov 03 00:48:46.675991 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:46.675937 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-bwb4h_48efe895-cacd-4880-94d4-3d965679e301/manager/0.log" Nov 03 00:48:47.709823 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:47.709789 5114 scope.go:122] "RemoveContainer" containerID="a9be850c3e5036bef3671cbce8417f8d9846818bd11d78308c2e86a2a91cfc15" Nov 03 00:48:47.723427 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:47.722754 5114 scope.go:122] "RemoveContainer" containerID="19106c0982e79c3adb3c0a9ff051c61b90f99f177a33bc1394586494b861bef6" Nov 03 00:48:47.736821 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:47.736789 5114 scope.go:122] "RemoveContainer" containerID="c24e39025374b0a7a6ec230f33e9b66e47f49178ab1159510082828ef0c0a00e" Nov 03 00:48:50.435738 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:50.435699 5114 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t6hpv/perf-node-gather-daemonset-t6krs"] Nov 03 00:48:50.440418 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:50.440387 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t6krs" Nov 03 00:48:50.529121 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:50.529085 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b82c345-1999-430d-8e7c-d69e4b5450c6-lib-modules\") pod \"perf-node-gather-daemonset-t6krs\" (UID: \"4b82c345-1999-430d-8e7c-d69e4b5450c6\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t6krs" Nov 03 00:48:50.529408 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:50.529381 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b82c345-1999-430d-8e7c-d69e4b5450c6-podres\") pod \"perf-node-gather-daemonset-t6krs\" (UID: \"4b82c345-1999-430d-8e7c-d69e4b5450c6\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t6krs" Nov 03 00:48:50.529584 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:50.529458 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b82c345-1999-430d-8e7c-d69e4b5450c6-sys\") pod \"perf-node-gather-daemonset-t6krs\" (UID: \"4b82c345-1999-430d-8e7c-d69e4b5450c6\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t6krs" Nov 03 00:48:50.529668 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:50.529606 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b82c345-1999-430d-8e7c-d69e4b5450c6-proc\") pod \"perf-node-gather-daemonset-t6krs\" (UID: \"4b82c345-1999-430d-8e7c-d69e4b5450c6\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t6krs" Nov 03 00:48:50.529668 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:50.529654 5114 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kvbk\" (UniqueName: \"kubernetes.io/projected/4b82c345-1999-430d-8e7c-d69e4b5450c6-kube-api-access-2kvbk\") pod \"perf-node-gather-daemonset-t6krs\" (UID: \"4b82c345-1999-430d-8e7c-d69e4b5450c6\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t6krs" Nov 03 00:48:50.530984 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:50.530959 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t6hpv/perf-node-gather-daemonset-t6krs"] Nov 03 00:48:50.630656 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:50.630618 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b82c345-1999-430d-8e7c-d69e4b5450c6-lib-modules\") pod \"perf-node-gather-daemonset-t6krs\" (UID: \"4b82c345-1999-430d-8e7c-d69e4b5450c6\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t6krs" Nov 03 00:48:50.630862 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:50.630688 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b82c345-1999-430d-8e7c-d69e4b5450c6-podres\") pod \"perf-node-gather-daemonset-t6krs\" (UID: \"4b82c345-1999-430d-8e7c-d69e4b5450c6\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t6krs" Nov 03 00:48:50.630862 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:50.630749 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b82c345-1999-430d-8e7c-d69e4b5450c6-sys\") pod \"perf-node-gather-daemonset-t6krs\" (UID: \"4b82c345-1999-430d-8e7c-d69e4b5450c6\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t6krs" Nov 03 00:48:50.630862 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:50.630776 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b82c345-1999-430d-8e7c-d69e4b5450c6-lib-modules\") pod \"perf-node-gather-daemonset-t6krs\" (UID: \"4b82c345-1999-430d-8e7c-d69e4b5450c6\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t6krs" Nov 03 00:48:50.630862 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:50.630824 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b82c345-1999-430d-8e7c-d69e4b5450c6-proc\") pod \"perf-node-gather-daemonset-t6krs\" (UID: \"4b82c345-1999-430d-8e7c-d69e4b5450c6\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t6krs" Nov 03 00:48:50.630862 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:50.630849 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b82c345-1999-430d-8e7c-d69e4b5450c6-podres\") pod \"perf-node-gather-daemonset-t6krs\" (UID: \"4b82c345-1999-430d-8e7c-d69e4b5450c6\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t6krs" Nov 03 00:48:50.631153 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:50.630869 5114 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kvbk\" (UniqueName: \"kubernetes.io/projected/4b82c345-1999-430d-8e7c-d69e4b5450c6-kube-api-access-2kvbk\") pod \"perf-node-gather-daemonset-t6krs\" (UID: \"4b82c345-1999-430d-8e7c-d69e4b5450c6\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t6krs" Nov 03 00:48:50.631153 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:50.630828 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b82c345-1999-430d-8e7c-d69e4b5450c6-sys\") pod \"perf-node-gather-daemonset-t6krs\" (UID: \"4b82c345-1999-430d-8e7c-d69e4b5450c6\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t6krs" Nov 03 00:48:50.631153 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:50.630889 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b82c345-1999-430d-8e7c-d69e4b5450c6-proc\") pod \"perf-node-gather-daemonset-t6krs\" (UID: \"4b82c345-1999-430d-8e7c-d69e4b5450c6\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t6krs" Nov 03 00:48:50.691833 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:50.691746 5114 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kvbk\" (UniqueName: \"kubernetes.io/projected/4b82c345-1999-430d-8e7c-d69e4b5450c6-kube-api-access-2kvbk\") pod \"perf-node-gather-daemonset-t6krs\" (UID: \"4b82c345-1999-430d-8e7c-d69e4b5450c6\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t6krs" Nov 03 00:48:50.753599 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:50.753564 5114 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t6krs" Nov 03 00:48:51.149172 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:51.149136 5114 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t6hpv/perf-node-gather-daemonset-t6krs"] Nov 03 00:48:51.153405 ip-10-0-135-250 kubenswrapper[5114]: W1103 00:48:51.153367 5114 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4b82c345_1999_430d_8e7c_d69e4b5450c6.slice/crio-2e1229d1eedb8290e8e5e59594c55ed590d9741c4cdfa335289fa5b009cca7d5 WatchSource:0}: Error finding container 2e1229d1eedb8290e8e5e59594c55ed590d9741c4cdfa335289fa5b009cca7d5: Status 404 returned error can't find the container with id 2e1229d1eedb8290e8e5e59594c55ed590d9741c4cdfa335289fa5b009cca7d5 Nov 03 00:48:51.315008 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:51.314953 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-d6fmn_289a684e-75c9-4b50-ad59-0ab341e55c61/manager/0.log" Nov 03 00:48:51.414297 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:51.414178 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t6krs" event={"ID":"4b82c345-1999-430d-8e7c-d69e4b5450c6","Type":"ContainerStarted","Data":"8a416402021f2dca753979970020a59a00730c3f3fd212b1344cfb6edf528dda"} Nov 03 00:48:51.414297 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:51.414278 5114 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t6krs" event={"ID":"4b82c345-1999-430d-8e7c-d69e4b5450c6","Type":"ContainerStarted","Data":"2e1229d1eedb8290e8e5e59594c55ed590d9741c4cdfa335289fa5b009cca7d5"} Nov 03 00:48:51.414532 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:51.414310 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t6krs" Nov 03 00:48:51.531571 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:51.531515 5114 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t6krs" podStartSLOduration=1.531496261 podStartE2EDuration="1.531496261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:51.530172027 +0000 UTC m=+7206.883205403" watchObservedRunningTime="2026-11-03 00:48:51.531496261 +0000 UTC m=+7206.884529635" Nov 03 00:48:53.451418 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:53.451382 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-bwb4h_48efe895-cacd-4880-94d4-3d965679e301/manager/0.log" Nov 03 00:48:57.439758 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:57.439722 5114 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t6krs" Nov 03 00:48:57.670521 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:57.670464 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-d6fmn_289a684e-75c9-4b50-ad59-0ab341e55c61/manager/0.log" Nov 03 00:48:58.236136 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:58.236100 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-sr5f2_0c414822-2ac4-4cd7-844f-a1f334dba25a/kube-state-metrics/0.log" Nov 03 00:48:58.645736 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:58.645706 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-sr5f2_0c414822-2ac4-4cd7-844f-a1f334dba25a/kube-rbac-proxy-main/0.log" Nov 03 00:48:59.064736 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:59.064694 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-sr5f2_0c414822-2ac4-4cd7-844f-a1f334dba25a/kube-rbac-proxy-self/0.log" Nov 03 00:48:59.811439 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:48:59.811353 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-bwb4h_48efe895-cacd-4880-94d4-3d965679e301/manager/0.log" Nov 03 00:49:00.334625 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:49:00.334596 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pw6nq_b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d/node-exporter/0.log" Nov 03 00:49:00.744957 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:49:00.744876 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pw6nq_b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d/kube-rbac-proxy/0.log" Nov 03 00:49:01.172093 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:49:01.172065 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pw6nq_b6b3abd8-5242-4bfd-bea9-9d4e0f4c4d4d/init-textfile/0.log" Nov 03 00:49:05.454365 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:49:05.454328 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65dbc8865f-58bxj_247dce35-9881-472f-b000-686c2a0a5dfa/kube-auth-proxy/0.log" Nov 03 00:49:06.106210 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:49:06.106173 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-7zbtz_19ba6ae1-0768-4641-9026-bce647422531/istio-proxy/0.log" Nov 03 00:49:06.529030 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:49:06.528930 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-92jjh_b4e963bb-8c2f-4628-a009-ebbebd62394f/istio-proxy/0.log" Nov 03 00:49:07.072574 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:49:07.072540 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-ksnhk_713030b6-6b1c-4cf9-b01a-58171f6ad0c6/istio-proxy/0.log" Nov 03 00:49:07.587000 ip-10-0-135-250 kubenswrapper[5114]: E1103 00:49:07.586959 5114 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qzjx" containerName="extractor" Nov 03 00:49:08.387153 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:49:08.387067 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-p266m_94071069-300b-43de-87e3-58e31ac8ab9d/prometheus-operator/0.log" Nov 03 00:49:08.802339 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:49:08.802311 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-p266m_94071069-300b-43de-87e3-58e31ac8ab9d/kube-rbac-proxy/0.log" Nov 03 00:49:09.223088 ip-10-0-135-250 kubenswrapper[5114]: I1103 00:49:09.223010 5114 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-8lzdt_ad69acdc-e65a-4aad-ac33-354f6ea07401/prometheus-operator-admission-webhook/0.log"